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Abstract 


This  study  introduces  a  pragmatic  methodology  for  examining  the  effectiveness  of  decision 
support  information  systems  for  Network  Centric  Operations  based  on  the  concept  of 
manufacturing  information  products.  We  first  develop  the  definitions  and  framework  for 
this  manufacturing  environment,  establishing  a  connection  with  previous  research  and 
extending  it  into  a  value-focused  domain.  We  then  apply  this  framework  to  elements 
extracted  from  two  recent  case  studies  involving  US  Forces  focused  on  validating  the  tenets 
of  Network  Centric  Warfare,  illustrating  its  unique  ability  to  identify  systems  level  concerns 
overlooked  previously.  Lastly,  we  extend  our  concern  for  the  reliability  of  information 
products,  showing  how  this  both  complements  a  quality  analysis  and,  taken  together, 
prescribes  a  methodology  for  creating  effective  information-based  decision  support  tools. 
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1.  Introduction 


If  one  is  willing  to  believe  that  devices,  personnel,  and  systems  deployed  and  operational  on  the 
modern  battlefield  are  perfect  and  that  users  get  exactly  what  the  information  they  want,  when  they 
want  it,  and  in  the  form  they  need  it  from  such  systems,  then  there  is  no  practical  reason  to  be 
concerned  with  information  quality  or  information  reliability.  Believing  thus  is  ill  advised,  however. 
Military  consumers  of  modem  information  have  by-and-large  woken  to  the  realization  that 
...information  abundance  does  not  necessarily  give  us  certainty.  Sometimes  it  can 
lead  to  errors  in  decision-making  with  undesirable  outcomes  due  to  either 
overwhelming  and  confusing  situations,  or  a  sense  of  overconfidence  leading  to 
improper  information  use.  The  former  situation  can  be  an  outcome  of  the  limited 
capacity  of  the  human  mind  in  some  situations  to  deal  with  the  complexity  and 
information  abundance;  whereas  the  latter  can  be  attributed  to  a  higher  order  of 
ignorance,  called  the  ignorance  of  self-ignorance.  [18] 

With  military  information  systems,  our  concern  principally  lies  with  the  latter  situation,  believing 
as  we  do  that  a  first  line  of  defense  against  uncertainty  lies  in  examining  military  information  systems 
from  a  multitude  of  perspectives  with  the  goal  of  developing  a  complementary  understanding  of 
system  capabilities  and  limitations.  The  conceptual  framework  introduced  by  Perry  et  al.  [23],  whose 
structured  perspective  on  information  quality  under  the  auspices  of  exploring  information  superiority, 
is  one  such  valuable 
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perspective,  having  led  to  what  is  currently  known  as  the  Network  Centric  Operations  Conceptual 
Framework  (Figure  1).  Portions  of  what  we  explore  herein  represent  refinements  and  extensions  of  a 
subset  of  their  constructs.  In  particular,  we  are  directly  concerned  with  issues  underlying  the  quality 
of  organic  and  individual  information  and  the  appropriate  structures  and  models  that  should  be  in- 
place  to  achieve  high  levels  of  this  quality. 

Our  intention  is  to  complement  their  work  with  a  concept  that  supports  the  design  and  analysis  of 
decision  support  products,  in  particular  common  operating  pictures  (COP),  and  uncovers  valuable 
insights  worth  incorporating  into  our  considerations  of  system  design  as  well.  Our  belief  is  that  for 
any  decision  support  information  system  deployed  in  a  Network  Centric  Operations  context,  a 
comprehensive  systems  design  addressing  quality,  reliability,  and  maintainability  of  information 
products  should  be  conducted.  Without  such  design  considerations,  successful  support  of  critical 
operational  elements  engaged  in  complex  operating  conditions  is  left  to  ad  hoc  innovation, 
serendipity,  and  hope;  which  is  not  a  strategy  for  success. 

1. 1  Why  a  product  manufacturing  focus? 

Managers  of  decision  support  information  systems  deployed  in  support  of  NCO  have  a  choice. 

They  can  either  view  information  as  a  byproduct  of  the  system  or  event,  or  as  a  product.  Wang  et  al. 
[29]  propose  five  common  factors  one  can  apply  to  determine  which  approach  has  been  adopted  by  an 


organization.  Table  1  shows  an  application  of  these  factors  in  consideration  of  NCO  systems. 


Information  as  a  Product 

Information  as  a  By-product 

What  is  managed? 

>  Information  product 

>  Information  Product  Life  Cycle 

>  Information  manufacturing  processes 

>  Hardware  and  software 

>  Systems  life  cycle 

How  is  it  managed? 

>  Integrated  cross-functional  approach 

>  Supply  chain  basis  encompassing 
information  suppliers,  manufacturers,  and 
end  users 

>  Integrate  stovepipe  systems 

>  Control  of  individual  components 

>  Cost  controls 

Why  is  it  managed? 

>  Deliver  quality  information  products  to 
end  users  with  high  service  reliability 

>  Implement  quality  hardware  and 
software  systems  with  high  system 
reliability 

What  is  success? 

>  Quality  information  products  continuously 
delivered  over  the  product  life  cycle  with 
high  service  reliability 

>  The  system  works  according  to 
required  specifications 

>  No  bugs 

>  Short-term,  “sustain”  perspective 

Who  manages  it? 

>  CIO 

>  Information  product  managers 

>  CIO 

>  IT  director,  database  and  system 
administrators 

Table  1.  Five  factors  to  determine  an  organization’s  approach  to  managing  information. 


One  way  of  understanding  the  difference  being  described  has  to  do  with  focus .  When  information 
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is  viewed  as  a  by-product,  the  management  focus  is  on  keeping  critical  systems  operating  in  support 
of  the  mission.  The  assumption  is  that  if  the  system  is  doing  what  it  is  expected  to  be  doing,  i.e.  it  is 
running,  it  has  not  been  compromised,  and  the  intended  end  users  and  input  sources  are  connected  as 
planned,  then  the  information  flowing  on  the  system  will  necessarily  be  acceptable.  This  is  not  a 
naive  or  uninformed  focus;  its  attention  is  merely  not  on  the  entity  that  is  actually  moving  from 
source  to  end  user. 

Conversely,  when  the  focus  is  on  the  entity  itself  and  information  is  viewed  as  a  product,  in 
particular  a  product  that  is  intentionally  manufactured  by  a  system,  a  host  of  new  concerns  arise  that 
exist  on  an  altogether  different  conceptual  layer  of  the  system.  The  activities,  or  processes ,  that  exist 
in  support  of  this  layer  are  at  least  as  critical  to  mission  success  as  the  physical  system  layer. 

The  five  factors  in  Table  1  serve  as  a  useful  diagnostic  for  any  organization  wishing  to  understand 
how  they  treat  information.  Additionally,  an  organization  can  easily  apply  a  very  simple  assessment 
to  determine  if  there  is  an  immediate  need  to  shift  their  management  focus  onto  the  flowing  entity: 

If  all  hardware  and  software  systems  are  successfully  working  as  described  in  Table  1, 
yet  end  users  are  still  employing  ad  hoc  <( work-arounds  ”  to  the  formal  system  processes 
in  order  to  obtain  information  they  need  to  accomplish  their  mission,  then  management 
must  shift  their  focus  onto  information  as  a  product  to  improve  information  quality 
levels. 

The  first  advocates  for  a  comprehensive  information  product  manufacturing  framework  for 
organizations  to  manage  their  data  and  information  handling  processes  was  Ballou  et  al.  [31].  Their 
approach  is  one  closely  linked  with  data  quality  management  in  the  sense  that  their  framework 
provided  a  methodology  for  tracking  data  units  from  source  to  inclusion  in  a  final  information 
product  delivered  to  an  end  user.  Two  effective  quantitative  measures  for  the  timeliness  criteria  of 
information  quality  were  introduced  in  their  work:  currency  and  volatility,  both  of  which  become 
important  elements  in  our  approach  as  well,  although  not  taken  together  in  this  manner. 

Our  approach  assumes  a  different  perspective,  preferring  to  remain  at  a  slightly  higher  level  of 
abstraction  for  pragmatic  reasons  related  to  systems  design  which  we  explain  later.  For  example,  they 
define  information  manufacturing  as  the  process  that  transforms  a  set  of  data  units  into  information 
products,  whereas  we  incorporate  the  underlying  product  purpose  into  our  definition.  Additionally, 
because  our  concerns  extend  into  the  lifecycle  of  dynamic  information  products,  we  must  necessarily 
be  concerned  with  structuring  service  reliability,  information  reliability,  and  maintainability  of 
information  products  into  our  framework  as  well  as  quality  concerns. 
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2.  Placement  within  a  systems  modeling  spectrum 


The  term  system  originates  from  the  Greek  systema  that  means  an  organized  whole.  The  term 
system  science  is  usually  associated  with  observations,  identification,  description,  experimental 
investigation,  and  theoretical  modeling  and  explanations  that  are  associated  with  natural  phenomena 
in  fields  [18].  More  often  than  not,  adopting  a  systems  perspective  on  collections  of  interacting 
elements  illuminates  holistic  qualities  easily  overlooked  by  invoking  a  Western  reductionism  [41] 
viewpoint  in  which  the  “whole  is  the  sum  of  its  parts”  [40].  Holism  assumes  that  the  “whole  is 
greater  than  the  sum  of  its  parts.”  For  systems  of  interacting  components,  a  holistic  philosophy 
contends  that  there  exists  emergent  behavior  in  tandem  activity  that  is  not  present  when  components 
are  acting  alone.  For  NCO  systems,  effects  such  as  compressed  parallel  mission  planning  activities 
afforded  by  digital  networked  communications  is  an  example  of  emergent  behavior  of  this  type. 
These  effects  are  commonly  referred  to  as  synergistic  effects.  The  philosophy  of  holism  supports 
how  we  conceptualize  information  as  an  entity. 


Automation  Routine 

of  Decision 

Decisions  Support 


Representing  Representing 

possible  insights  for 

system  debate 

designs  and 
changes 


Tools  for  routine  decision  Tools  for  thinking 

making 

Figure  2.  Placement  of  study  within  a  systems  modeling  spectrum  [26]. 


Definition.  A  systems  perspective  of  an  information  product  is  as  a  collection  of  individual 
components  whose  support  to  purpose  is  greater  as  an  assembled  manufactured  information 
product  than  any  of  the  individual  components  could  provide  when  taken  individually. 

We  view  an  information  product  as  a  system  in  a  manner  distinct  from  the  technological  system 
that  houses  and  disseminates  this  product.  Stated  another  way  using  a  concrete  example:  a  properly 
manufactured  COP  (information  product)  provides  more  direct  support  for  effective  decision-making 
than  any  of  the  individual  elements  comprising  the  COP  could  do  when  taken  alone.  The  whole  is 
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greater  than  the  sum  of  its  parts. 

We  also  adopt  a  construction  or  compositional  emergence  epistemic  position  [42]  with  regards  to 
designing  information  products  for  manufacturing.  This  is  a  form  of  microdeterminism  in  which  the 
parts  and  their  interactions  comprise  the  structure  of  the  larger  system.  This  implies  that  before  a 
cohesive  information  product  can  be  designed  and  manufactured,  one  has  to  first  understand  how  the 
components  of  this  product  relate  to  the  unifying  purpose  individually,  the  type  of  inter-component 
support  relationships  they  exhibit  (positive,  neutral,  negative),  and  what  type  of  holistic  behavior  they 
should  exhibit  when  assembled  properly. 

Figure  2  illustrates  the  placement  of  this  study  and  its  content  within  the  spectrum  of  possible 
systems  modeling  efforts.  At  one  end  of  the  spectrum  are  models  that  are  designed  for  routine 
decision  making.  These  models  typically  replace  or  supplant  specific  functions  previously  performed 
by  human  decision-makers.  Good  examples  are  the  classification  and  identification  algorithms 
imbedded  in  unattended  ground  sensor  cluster  gateway  nodes  [43].  They  routinely  ‘decide’  on  what 
they  are  detecting  using  feature  set-based  mathematical  models  built  into  their  software. 

At  the  other  end  of  the  spectrum  are  models  whose  purpose  are  to  enhance  human  understanding; 
hence,  tools  for  thinking.  They  provide  a  lens  through  which  decision-makers  can  view  complex  and 
challenging  problems  or  the  environments  that  contain  these  problems.  The  cognitive  framework 
reflected  in  a  ‘systems  perspective’  is  a  classic  example  of  one  such  model. 

This  study  resides  at  this  end  of  the  spectrum  as  well,  as  indicated  in  Figure  2.  It  proposes  a  model 
for  thinking  about  information  that  is  used  to  support  decision-making  on  the  battlefield:  as  an 
intentionally  manufactured  information  product  that  is  the  result  of  carefully  structured  and  controlled 
manufacturing  processes. 

2. 1  Systems  placement  of  the  manufacturing  metaphor 

The  concept  of  an  information  manufacturing  process  aligns  with  metaphors  principally  drawn 
from  cybernetics  that  emphasize  active  learning  rather  than  an  alternative  passive  adaptability  that 
characterizes  metaphors  that  see  the  information  network  and  the  humans  interacting  with  it  as  an 
organism  [20].  Thus,  attention  is  focused  on  decision-making,  information  processing  and  control. 
Underlying  the  information  manufacturing  metaphor  is  an  assumption  that  the  organization  has 
decided  on  a  product’s  unifying  purpose  and  is  challenged  to  design  complex  systems  to  respond  to 
surrounding  turbulent  environments  the  product  finds  itself  placed  within  upon  completion  and 
distribution. 

While  an  information  manufacturing  system  can  be  conveniently  represented  as  a  simple  flow 
diagram,  one  recognizes  that  a  large  degree  of  control  over  the  product  is  decentralized  because 
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induced  change  is  sufficiently  complex  to  preclude  it  from  being  handled  exclusively  at  the  top  of  the 
organization.  The  organization  must  manage  single-loop  learning,  correcting  deviations  from 
prescribed  quality  criteria  goals.  At  the  same  time,  it  must  also  be  capable  of  double-loop  learning, 
changing  the  nature  of  its  purposes  if  these  become  unattainable  as  the  environment  shifts  [21]. 
Military  operations  have  a  natural  affinity  to  this  type  of  metaphor  as  the  requirement  for  active 
learning  is  a  prerequisite  for  successful  leaders,  units,  and  missions. 

The  information  manufacturing  metaphor  does  not  address  the  motivation  of  individuals  acting  as 
imbedded  elements  of  the  process,  to  conflicting  elements  of  interactions,  nor  how  fundamental 
purposes  are  actually  derived.  These  shortcomings  do  not  detract  from  the  effectiveness  of  the 
metaphor  in  getting  at  information  quality  in  a  military  setting  because  of  the  homogeneous  nature  of 
these  elements  in  military  operations.  Individual  motivations  are  united  to  achieve  the  mission, 
conflicts  are  subservient  to  command,  and  purposes  are  clearly  reflected  in  mission  intent. 

3.  The  nature  of  data,  information,  and  knowledge 

Data  is  raw,  unstructured  bits  of  quantitative  and/or  qualitative  facts.  For  example,  the  number 
“210”  and  the  words  ‘fast’  and  “agitated”  are  just  facts.  Structured  data  by  purposeful  design  or  in 
response  to  system  user  queries  is  information.  Structuring  the  three  pieces  of  data  in  response  to  the 
user  queries,  “How  heavy  is  the  boxer?”,  “Is  he  fast  or  slow?”,  and  “What  is  his  demeanor?”  changes 
the  three  bits  of  data  to  contextualized  formatted  data,  or  information.  When  merged  with  the  user’s 
experience  of  boxing  opponents  fitting  these  characteristics,  the  information  becomes  actionable 
knowledge. 

Researchers  [18]  posit  that  knowledge  is  created  at  three  distinct  levels:  cognitive,  knowledge  that 
can  be  acquired  by  the  senses;  knowledge  based  on  correct  reasoning  from  hypotheses;  and  belief 
followed  by  conjecture,  in  which  knowledge  is  based  on  inference,  prediction,  or  theorization  based 
on  incomplete  or  reliable1.  For  information  products,  the  various  forms  of  data  serve  as  input 
resource  streams  to  the  process  of  manufacturing,  as  do  labor,  materials,  time  and  other  typical 
manufacturing  elements. 

Data  quality  efforts,  while  sharing  some  of  the  characteristics  of  information  quality,  seek  to  reduce 
errors,  meet  specifications,  and  somewhat  in  common  with  a  service  reliability  standpoint,  increase 
customer  satisfaction.  In  contrast,  information  quality  has  the  goal  of  improving  the  usefulness  and 
validity  of  information,  in  this  case,  information  products.  It  is  inherently  a  system-level  focus,  one 
that  assumes  a  posture  of  insuring  fundamental  quality  characteristics  are  explicitly  considered  during 


1  We  use  the  term  ‘reliable’  here  in  a  sense  consistent  with  reliability  theory  of  knowledge  [19]. 
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the  creation,  deployment,  and  maintenance  of  both  information  products  and  their  associated 
manufacturing  processes  in  a  network. 

Information  serves  purposes  that  are  determined  by  an  individual  user  or  groups  of  homogenous 
users  sharing  common  purposes.  In  general,  information  communicates  answers  to  questions  and 
statements  about  situations  and  facts.  When  information  is  created  and  shared  over  digital  networks 
for  the  purpose  of  decision  making,  we  refer  to  these  digital  networks  as  decision  support  information 
systems  (DSIS).  Sensor-to-shooter  targeting  systems  are  an  example  of  one  subset  of  DSIS  in  which 
a  fire/no  fire  decision  must  inevitably  be  made. 

The  domain  of  knowledge  management  is  principally  concerned  with  recording,  storing,  and 
making  readily  accessible  to  appropriate  users  vital  knowledge  elements  that  sustain  the  core  of 
operational  identity  of  an  organization.  The  resulting  knowledge  base  consists  of  both  knowledge 
about  key  processes  (know  how)  and  knowledge  within  these  processes  (know-what).  Information 
managers  concern  themselves  with  the  proper  selection,  structuring,  and  delivery  of  information 
products  to  users  in  response  to  needs.  Data  managers  focus  on  the  acquisition,  storage,  and 
transmission  of  raw  data  as  an  input  resource  to  information  products,  principally  those  used  in 
support  of  decision  making.  Issues  of  security  are  shared  in  all  three  layers.  Information  quality 
underscores  the  knowledge  base. 

In  order  to  assess  and  possibly  improve  the  quality  of  information  in  NCO  systems,  focus  should  be 
applied  to  both  the  content  and  structure  of  the  information  products  [6],  the  manufacturing  processes 
(manipulation,  service  to  end  users),  and  the  system  infrastructure.  The  latter  of  these  remains  a 
concern  of  computer  science  and  information  technology  specialists  and  will  not  be  directly  addressed 
in  this  paper. 

3. 1  The  value  of  information 

In  this  study,  our  interest  is  not  in  assessing  or  managing  the  value  of  information  to  network 
centric  organizations.  However,  it  is  worthwhile  to  briefly  discuss  several  important  results  in  this 
area  as  they  pertain  to  information  so  that  the  distinction  between  quality  and  value  is  apparent. 

Over  time,  researchers  have  proposed  analytical  approaches  for  assessing  the  value  of  information, 
which  are  quite  apart  from  the  notion  of  quality,  reliability,  or  maintainability  of  information 
products.  The  most  predominant  concept  is  an  extension  of  the  thermodynamics  principle  involving 
entropy. 

Entropy  is  a  property  of  all  thermodynamic  systems  and  is  represented  by  the  following  equation: 

A s>JS., 

T 

where  AS  is  the  change  in  entropy,  8Q  is  the  change  in  heat  energy  and  T  is  some  constant 
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temperature.  Whenever  heat  enters  or  leaves  a  system,  entropy  changes.  Notice  that  the  change  in 
entropy  is  positive  for  every  interaction,  meaning  that  entropy  for  part  of  a  thermodynamic  system 
can  be  reduced,  but  entropy  as  a  whole  for  the  system  increases.  The  only  way  that  entropy  can  be 
reduced  overall  is  by  putting  energy  into  a  system.  It  is  this  property  that  points  to  an  analogy  with 
uncertainty  -  it  seems  to  increase  of  its  own  volition  unless  something  is  actively  done  to  reduce  it 
(energy). 

Brillouin  [47]  introduced  Schrodinger’s  idea  of  negative  entropy,  or  negentropy ,  replacing  S  with 
an  expression  of  opposite  sign  N=-  S,  which  behaves  more  intuitively  than  entropy,  steadily 
decreasing  over  time  [48].  This  concept  provided  a  link  with  both  information  theory,  relating 
uncertainty  to  entropy  and  information  to  negentropy.  As  more  information  (quantity)  is  gained 
about  a  system,  both  uncertainty  and  entropy  decrease. 

The  seminal  work  by  Shannon  [32]  and  its  associated  derivatives  directly  exploit  this  association 
but  intentionally  exclude  considerations  of  the  semantic  meaning  of  the  message.  Shannon’s  concept 
of  mutual  information ,  which  characterizes  information  value  in  terms  of  entropy  functions,  was 
originally  associated  with  analyzing  the  source  of  the  message  to  discover  what  capacity  is  required 
in  a  communications  channel  to  transmit  all  the  messages  that  the  source  provides,  and  then  extended 
to  consider  information.  This  extension  is  based  on  an  assumption  that  the  uncertainty  regarding  any 
variable  Z  characterized  by  a  probability  distribution  f{z)  can  be  represented  by  the  entropy 
function: 

//(z)  =  -£/(z)log/(z) 

z 

An  example  is  helpful  [49].  Suppose  that  we  were  interested  in  some  target  variable  T  whose  value 
has  some  amount  of  total  uncertainty  H(T )  associated  with  it.  Let  A  represent  the  total  possible  set 
of  information  that  we  could  acquire  to  help  resolve  this  uncertainty.  Then,  the  residual  uncertainty 
regarding  the  true  value  of  T7,  given  that  we  acquire  x,  is  then  written  as  an  entropy  expression 

H(T\x)  =  -Jjf(t\x)\ogf(t\x), 

t 

The  average  residual  uncertainty  in  T7,  summed  over  all  possible  information  we  could  acquire  x  can 
then  be  written  as 

H{T\X)  =  yjH{T\x)f{x) 

X 

-?  m 

The  total  uncertainty  reducing  potential  of  A,  called  Shannon’s  mutual  information,  is  given  by 

i(t\x)=h(t)-h(t\x ) 
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There  are  two  main  limitations  with  this  approach.  First  is  that  there  is  no  notion  of  the  relevance 
to  the  task  at  hand  [36].  Viewing  two  messages  from  distinct  sources  as  equivalent  because  they  have 
the  same  entropy  value  misses  the  point  that  when  they  originate  in  different  situations  their  value 
could  be  different.  Moreover,  representing  the  loss  of  information  in  terms  of  entropy  functions  fail 
to  directly  accommodate  notions  of  quality  with  respect  to  information  products. 

Secondly,  Shannon’s  measure  does  not  reflect  ordering  or  scale  information  relative  to  the  values 
that  a  variable  can  take  on.  Consequently,  while  it  does  measure  the  cost  of  resolving  uncertainty,  it 
cannot,  for  example,  differentiate  between  the  cost  of  obtaining  critical  information  and  non-critical 
information  in  doing  so. 

Overall,  this  value  focus  concentrates  on  the  process  of  communicating  and  obtaining  information 
and  the  associated  loss  or  gain  in  value  as  a  result,  a  task  we  propose  is  more  naturally  assigned  to 
information  assurance  efforts  than  quality  assessments.  As  a  side  note,  Barr  and  Sherrill  [33] 
extended  a  portion  of  this  earlier  work  to  encompass  tactical  information.  Again,  however,  the  notion 
of  assessing,  maintaining  and  sustaining  information  quality  are  not  present  in  their  approach. 

Glazer  [34]  advanced  the  discussion  of  information  value  to  a  more  systems  level  approach, 
characterizing  behavioral  priorities  in  information-intensive  firms  that  lead  to  real  competitive 
advantage.  His  underlying  point  is  one  that  we  take  to  heart  in  this  study  as  well:  information¬ 
intensive  organizations  (transformed  US  Forces)  see  Information  Technology  (IT)  as  the  enabler  of 
growth  in  the  production  and  distribution  of  information.  The  output  of  these  technological  systems, 
the  information  itself  carries  value.  His  approach  has  received  a  small  amount  criticism  because  it 
ignores  the  value  added  element  associated  with  the  technology  [35]. 

Most  importantly  for  our  purposes,  he  argues  that  successful  organizations  have  gone  beyond  a 
focus  on  the  technology  to  view  the  information  itself  as  the  organization’s  primary  strategic  asset 
and  have  directed  management  to  focus  their  attention  on  this  key  variable.  Using  a  product  for 
purchase  transaction  as  the  base  unit  of  analyses,  he  identified  three  general  components  of 
information  value: 

1 .  Given  the  information,  revenues  from  subsequent  transactions  are  greater  than  they 
otherwise  would  be; 

2.  Given  the  information,  costs  of  subsequent  transactions  are  lower  than  they  otherwise  would 
be;  and, 

3.  The  information  itself  is  marketed  and  sold  to  other  firms. 

The  firms  he  examines  have  a  good  deal  in  common  with  the  organizations  envisioned  for  a 
transformed  US  Force.  Defining  a  transaction  as  an  exchange  of  information  product(s)  that  occurs 
between  the  information  manufacturing  system’s  application  layer  and  the  end  user,  we  can  translate 
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and  restate  his  three  general  components  of  value  for  use  with  NCO  systems 


Definition.  For  NCO  systems,  indications  of  information  product  value  fall  into  three  assessment 
components: 

1.  Given  the  information  product,  mission  effectiveness  and  efficiency  is  greater  than  it 
otherwise  would  be; 

2.  Given  the  information  product,  mission  costs  (resources,  including  loss  of  life)  are  lower 
than  they  otherwise  would  be;  and, 

3.  The  information  products,  or  appropriately  tailored  versions  of  them,  are  marketed  and 
shared  with  multi-national  forces  in  a  manner  that  illuminates  a  growing  dependency  on 
these  products  by  these  forces. 

While  the  first  two  valuation  components  are  part  of  the  existing  NCOCF  thought  processes,  the 
third  is  new.  It  recognizes  that  in  a  multi-national  force  setting,  the  information  products  have  value 
apart  from  the  missions  they  are  designed  to  support. 

NCW  products  such  as  case  studies,  short  courses,  and  information  created  within  the  social 
networks  of  the  Transformation  Chair  network  are  products  of  this  type.  The  growing  demand  for 
these  products  outside  of  the  context  within  which  they  were  intended  to  serve  (e.g.,  capturing  lessons 
learned  and  observations  of  network-enabled  forces  during  the  Iraq  War)  demonstrate  the  existence  of 
such  a  market. 

The  choice  of  an  object  of  valuation  appears  to  some  researchers  [35]  to  be  a  prerequisite  for  the 
definition  of  the  value  of  information.  Arising  principally  in  economics  ([37],  [38],  [39]),  this 
concept  gave  birth  to  the  notion  of  an  information  commodity.  This  provided  a  conceptual  bridge 
between  the  general  abstraction  of  information  and  the  valuation  of  a  specific  manifestation  of 
information:  a  product.  And  once  an  object  can  be  validly  conceptualized  as  a  product,  all  of  the 
associated  metaphors  and  modeling  issues  associated  with  manufacturing  products  become  fair  game 
to  explore. 

Separating  the  furnishing  of  information  from  the  valuation  of  information  in  these  same  studies 
proved  another  useful  advance  that  we  build  upon  in  a  related  study  [44]  in  which  we  examine  the 
concept  of  service  reliability  as  it  relates  to  information  product  delivery.  Separating  product  quality 
from  service  reliability  is  an  important  distinction  to  make  because  it  effectively  assesses  a  decision 
support  information  system  (DSIS)  from  the  end  user  perspective  rather  than  the  system  design  and 
management  viewpoint. 
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4.  Information  product  manufacturing  perspective 

Much  of  the  previous  work  associated  with  information  quality  focuses  exclusively  on  data  quality 
([2],  [6])  or  attempts  to  frame  a  general  notion  of  information  characteristics  that,  while  important  to 
theoreticians,  is  lacking  in  its  ability  to  provide  guidance  as  to  how  one  might  go  about  imbuing  these 
characteristics  into  specific  user  required  products.  Data  quality  is  a  directly  measurable,  low-level 
construct.  It  is  relatively  easy  to  structure  metrics  against  criteria  of  data  quality  since  the  objects  in 
data  bases  can  be  tracked  back  to  actual  data  generation  sources  and  checked  directly  for  error 
presence,  missing  elements,  or  other  similar  quality  indicators  while  they  reside  in  storage. 

The  first  such  efforts  immerged  from  the  field  of  accounting  ([25],  [27]).  These  were  motivated 
over  concerns  about  the  integrity  of  the  data  being  used  for  both  corporate  performance  analysis  and 
actuarial  assessments  of  insurance  instruments.  Since  that  time,  pragmatic  differentiations  between 
data,  information,  knowledge,  and  wisdom  have  come  to  pass  that  have  afforded  researchers  the 
opportunity  to  fine  tune  there  conceptual  representations  [18]. 

Information,  being  a  higher-level  abstract  object,  is  a  good  deal  more  challenging  to  assess  in  any 
of  the  three  areas  noted:  quality,  reliability,  and  maintainability.  Doing  so  requires  careful 
consideration.  Although  many  of  the  concepts  associated  with  classical  theory  related  to  these  areas 
may  at  first  glance  appear  to  directly  apply,  we  are  cognizant  of  the  words  of  Cushing  [24],  one  of  the 
earliest  pioneers  interested  in  applying  reliability  measures  to  data  systems: 

“The  concepts  of  reliability  and  maintainability  are  both  borrowed  from  the  field  of 
engineering  for  application  in  a  context  far  removed  from  that  envisioned  by  their 
creators.  There  is  no  assurance  that  such  borrowed  concepts  will  be  meaningful  in  a 
new  environment,  and  it  is  therefore  incumbent  upon  the  borrower  to  demonstrate  the 
connection  between  the  concept  and  its  intended  application.  ” 

Pinning  our  focus  to  the  notion  of  an  information  product  is  a  practical  choice  that  helps  ameliorate 
this  challenge.  It  frames  information  quality  as  relative  to  something  in  particular.  Constructing  a 
framework  within  which  one  can  pragmatically  discuss,  design,  analyze  and  adapt  such  information 
products  is  empowering  and  particularly  appropriate  for  Network  Centric  systems. 

Information  is,  at  its  essence,  an  abstract  object  characterizing  anything  that  comes  to  us  by  way  of 
our  senses  that  through  cognitive  processes,  more  specifically  a  dialectical  process  [18],  informs  us. 

It  defines  both  our  interface  with  existence  and  our  conditioned  way  of  understanding  this  existence. 
We  shape  information,  trade  it,  hoard  it,  accumulate  it,  destroy  it,  and  store  it.  At  the  right  place  in 
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the  right  form  at  the  right  time,  we  find  it  immeasurably  valuable.  At  the  right  place  in  the  right  form 
at  the  wrong  time,  it  is  utterly  useless  to  us. 

4. 1  Manufacturing  Components  Defined 

Military  professionals  in  decision  making  roles  have  long  recognized  information’s  potential  to 
turn  the  tide  of  battle  or  at  least  afford  an  operational  advantage  when  its  possession  represents  an 
asymmetric  advantage  to  friendly  forces.  The  military  classification  system  is  testament  to  this 
assertion.  If  information  -  whatever  it  may  be  -  can  be  owned  and  valued,  it  can  be  a  commodity. 
Rather  than  speaking  of  information  as  a  commodity  per  se,  we  chose  to  call  it  an  information 
product. 

Definition.  An  information  product  is  an  intentionally  constructed  assemblage  of  static  and 
dynamic  components  bound  together  by  a  single  cohesive  purpose  for  an  audience  whose  needs  for 
information  are  known  and  well-understood. 

While  seemingly  broad  in  its  definition,  it  nonetheless  sufficiently  bounds  the  concept  so  that  one  can 
properly  access  notions  of  information  quality  and  service  reliability.  Yet,  it  remains  general  enough 
to  apply  to  all  forms  of  military  information  used  in  tactical,  operational,  and  strategic  decision¬ 
making. 


Definition.  The  organized  collection  of  activities  and  dynamic  flow  of  resources  that  results  in  the 
creation  of  an  information  product  is  an  information  manufacturing  process. 

Definition.  An  information  system  that  produces  predefined  information  products  is  an 
information  manufacturing  system. 

Our  appeal  to  the  concept  of  viewing  the  creation  and  dissemination  of  information  as 
manufacturing  information  products  extends  previous  work  in  this  area  ([6],  [34]).  Believing  that 
there  is  value  in  modeling  and  analyzing  information  apart  from  the  hardware  components  carrying  it, 
a  manufacturing  framework  serves  a  variety  of  complementary  purposes. 

First,  since  the  activity  of  manufacturing  is  familiar  to  most,  translating  and  applying  appropriate 
metaphors  and  concepts  to  information  is  both  natural  and  appealing  because  the  abstraction  is  simply 
accommodated.  We  do  not  go  so  far  as  to  suggest  that  all  quality-related  manufacturing  theory  and 
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practices  validly  translate  to  NCO  information  systems.  However,  we  strongly  believe  that  under  the 
correct  structural  framing  a  critical  subset  do,  and  those  that  do  offer  an  opportunity  to  revise,  restate, 
and  extend  elements  of  the  NCOCF.  Moreover,  we  note  that  fundamental  spatial  arrangements  of 
manufacturing  facilities  nicely  decompose  and  organize  the  three  major  elements:  data,  information 
products,  and  knowledge  into  an  appropriate  hierarchy  with  regards  to  military  information  systems. 

Secondly,  thinking  about  information  systems  in  this  manner  shines  a  spotlight  on  critical  systems 
issues  concerning  the  manufacturing  processes  themselves.  For  example,  where  should  quality 
blocks  (to  be  explained)  be  placed  in  the  process?  What  resource  acquisition  policies  should  be  in 
effect?  What  skill  sets  are  required  by  the  individuals  involved  in  the  manufacturing  processes?  How 
are  these  to  be  maintained  knowing  that  rotation  and/or  attrition  will  occur  among  these  personnel? 
What  raw  data  materials  should  go  into  the  product?  How  should  the  product’s  quality  levels  be 
maintained?  Which  end  user  requirements  are  fixed  and  which  evolve?  Who  should  create  the 
product’s  dynamic  and  static  components?  What  functions  should  these  perform?  Which  product 
presentation  form  is  most  effective  in  supporting  the  purpose?  Inevitably,  issues  such  as  these  are  tied 
to  quality  considerations. 

Interestingly,  even  more  subtle  issues  arise  as  second  order  effects  in  a  manufacturing  context.  In 
particular,  we  note  that  the  world  view  represented  in  the  manufactured  product  has  an  impact  on  its 
perceived  value  to  the  user.  Does  it  ultimately  represent  the  concerns  of  the  manufacturer,  the  end 
user,  or  both?  And,  if  the  pool  of  end  users  changes  over  time  as  it  most  assuredly  does,  can  the 
information  manufacturing  process  evolve  to  accommodate  and  align  with  the  needs  of  the  changing 
user  pool  as  well?  Or  must  end  users  develop  work-around  processes  to  maintain  the  product’s 
usefulness?  How  will  the  manufacturing  facility  identify  the  emergence  of  quality  degradation 
indicators?  What  are  these  indicators? 

A  manufacturing  perspective  naturally  motivates  design  concerns  that  focus  on  the  quality  of 
specific  content  elements.  For  information  products,  we  identify  nine  characteristics  of  quality 
information  that  should  be  present  in  every  product  released  to  an  end  user.  The  extent  to  which  each 
of  these  characteristics  is  present  and  exactly  how  a  designer  should  measure  each  one  of  them  is  still 
an  open  question,  although  we  investigate  a  select  number  of  options  in  this  paper. 

Starting  with  the  original  eight  elements  of  information  quality  stated  in  the  NCOCF,  Perry  et  al. 
[23]  carefully  construct  one  methodology  for  doing  this  for  the  notion  of  information  in  general  as  it 
applies  to  the  construction  of  what  they  refer  to  as  the  common  relevant  operating  picture  (CROP). 

We  take  the  process  further  by  both  widening  the  span  of  consideration  to  quality,  maintainability, 
and  reliability  in  this  study.  The  information  product  manufacturing  perspective  enables  us  to  tie 
these  three  areas  together  in  a  cohesive  manner,  thereby  beginning  to  construct  a  robust  and  novel 
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concept  of  information  reliability,  an  on-going  effort  whose  basic  elements  are  presented  in  the  final 
section  of  this  paper. 

4.2  Dynamic  Quality  Growth  Over  Time 

While  it  is  recognizably  important  to  comprehend  the  concept  of  information,  focusing  on  the 
manufacturing  of  information  products  creates  actionable  criteria  for  quality  that  can  guide  the 
creation  of  specific  product  features  and  the  processes  associated  with  creating  these  features. 

Treating  information  as  a  product  requires  four  basic  principles  be  followed  [29]: 

1 .  Understand  the  end  user’s  information  needs. 

2.  Manage  information  as  a  product  of  a  well-defined  production  process. 

3.  Manage  information  as  a  product  with  a  defined  life  cycle. 

4.  Appoint  an  information  product  manager  to  manage  the  information  manufacturing 
processes  and  resulting  product. 

Most  of  what  follows  explores  principles  2  -  3  in  detail.  The  fourth  principle  follows  directly  from 
this  exploration.  The  first  principle  requires  that  system  designers  clearly  understand  what  an  end 
user  needs  with  regards  to  information  supporting  decision-making,  when  it  will  be  needed,  and  in 
what  form  it  is  needed.  These  are  obvious.  Less  obvious  are  the  converse  of  this  principle:  we  must 
also  understand  what  an  end  user  does  not  need,  when  the  product  or  some  subset  of  its  components 
are  not  needed,  and  in  what  form  it  should  not  be  delivered.  It  is  this  complementary  principle  that 
offers  some  respite  to  the  logjam  of  information  overload  because  it  suggests  that  an  end  user  may  not 
need  everything  all-the-time  present  in  their  information  products.  This  is  very  important  in  what 
follows. 


Figure  3.  Command  21  Knowledge  Wall  conceptual  design  (source:  SPAWAR,  1999). 


Information  overload  can  be  diagnosed  by  observing  that  the  end  user  cannot  productively  use  the 
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quantity  of  information  within  the  time  scale  available.  The  right  amount  of  information  is  a  cautious 
construct  that  illustrates  that  the  criteria  of  comprehensiveness  and  completeness  with  regard  to 
information  are  relative  terms.  The  effectiveness  of  a  system  can  consequently  be  assessed  by 
measuring  the  “knowing  -  doing  gap”  [9]. 

We  posit  that  this  insight  presents  an  opportunity  to  design  dynamic  information  products  whose 
quality  levels  intentionally  fluctuate  to  meet  target  levels  of  quality  criteria  at  various  scheduled  times 
during  their  life  cycle;  a  just-in-time  strategy  with  appropriate  redundancy  in-place. 

For  clarity  on  this  point,  we  are  not  simply  proposing  that  products  are  timed  for  optimal  delivery 
to  the  end  user,  although  this  certainly  may  be  the  case.  We  are  proposing  that  a  single  information 
product  can  be  decomposed  into  interrelated  component  parts  not  all  of  which  need  to  be  at  high 
levels  of  quality  and  reliability  criteria  at  the  same  time.  Figure  3  illustrates  the  varied  number  of 
interrelated  components  that  can  and  do  appear  on  a  single  information  product.  As  we  later  describe, 
depending  on  the  predetermined,  time-dependent  needs  of  the  end  user  for  quality,  it  is  possible  to 
schedule  the  delivery  of  quality  at  various  levels  to  maximize  the  effectiveness  of  the  information 
product  as  it  relates  to  the  purpose  for  which  it  was  created,  and  to  minimize  information  overload  by 
not  providing  excessively  high  quality  components  when  they  are  not  needed. 

This  scheduled  variability  in  quality  criteria  levels  corresponds  to  a  “Big-Q”  -  “Little-q”  definitions 
of  quality. 

Definition.  A  Big-Q  notion  of  information  quality  recognizes  that  there  are  a  core  set  of  criteria 
that  are  required  of  all  information  products  prior  to  acceptance  by  an  end  user. 

Definition.  A  Little-q  notion  of  information  quality  establishes  the  degree  to  which  an  information 
product  meets  or  satisfies  its  underlying  Big-Q  quality  criteria. 

For  a  host  of  manufactured  products,  these  two  concepts  become  a  management  concern  at  the 
point  of  release  of  the  product.  The  typical  assumption  at  release  is  that  quality  monotonically  non¬ 
increases  past  this  point.  Most  attempts  at  quantifying  quality  criteria  for  information  in  fact  assume 
constant  deterioration  over  time,  independent  of  context,  a  point  over  which  we  take  exception. 

What  we  are  proposing  is  that  a  single  information  product  can  have  variable  Little-q  levels  of 
quality  criteria  over  its  planned  life  cycle.  The  manufacturing  process  in-place  should  be  flexible, 
adaptable,  and  responsive  enough  to  be  able  to  accommodate  this  planned  variability  through 
appropriate  quality  controls  inserted  into  the  progression  of  manufacturing  activities. 

This  focus  on  manufacturing  information  products  inherently  assumes  a  systems  perspective  ([21], 
[26])  that  enables  one  to  consider  information  as  an  aggregate  assemblage  of  components  whose 
collective  behavior  or  message  as  a  system  is  not  present  at  the  individual  component  level.  An 
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information  product  is  properly  viewed  as  a  system  under  a  specific  framework  that  we  introduce  in 
what  follows.  Understanding  the  nature  of  information  products  enhances  the  suite  of  modeling  tools 
available  to  NCO  systems  designers  to  create,  distribute,  protect,  and  maintain  these  products  over 
their  usable  lifetime. 

From  a  systems  perspective,  adopting  the  framework  of  information  product  manufacturing  creates 
natural  allegorical  structures  against  which  designers  can  innovate,  extract  critical  ingredients,  and 
consider  time-evolutionary  effects  such  as  service  erosion,  reliability  degradation,  and  maintenance  of 
quality. 

Furthermore,  as  will  be  explained,  an  information  product  manufacturing  framework  appears  to 
exclusively  enable  one  to  link  information  products  such  as  a  common  operating  picture  (COP)  to 
purpose ,  a  key  link  if  one  wishes  to  decide  which  information  components  should  be  included  or 
excluded  based  on  considerations  of  inference  chains  [28].  It  then  becomes  possible  to  examine  the 
reliability  of  the  COP  in  terms  of  either  goodness,  or  risk-to-use  (RTU)  based  on  the  degree  of 
inferential  support  the  purpose-networked  elements  provide  at  any  point  in  the  lifetime  of  the  product. 
This  particular  dimension  of  this  study  is  currently  under  investigation  by  the  authors. 

4.3  Dynamic  Information  Products  and  Quality 

Battlespace  factors  exogenous  to  information  systems,  such  as  weather,  soldier  stress,  cognitive 
error,  and  weapons  effects,  compound  the  challenges  for  decision  support  information  systems  to 
operate  as  idealized  in  their  design.  In  any  framework  addressing  quality,  reliability,  and 
maintainability,  it  is  prudent  to  view  the  products  of  such  systems  as  dynamic  as  well  as  static,  taking 
into  account  the  critical  components,  their  functions,  and  the  manufacturing  processes  associated  with 
these  products. 

A  reasonable  systems  scientist  would  suspect  that  some  information  quality  criteria  levels  would 
naturally  change  continuously  over  time  while  others  may  make  discontinuous  jumps  upwards  or 
downwards.  We  likewise  suspect  there  are  some  criteria  of  quality  that  are  relatively  time-invariant, 
yielding  to  other  yet-to-be-determined  influencing  factors  instead.  Failing  to  understand  how  these 
criteria  coexist,  how  they  dynamically  relate,  and  what  binds  them  together  into  a  unified  product, 
runs  the  risk  of  designing,  developing,  and  distributing  products  to  operational  units  that  are,  at  best, 
approximations  of  the  products  they  could  have  or  should  have  been.  Resolving  this  suspicion  is  an 
open  question  to-date. 

Considering  information  product  components  and  manufacturing  processes  in  a  dynamic  fashion 
likewise  suggests  an  interesting  possibility  we  mentioned  earlier:  that  information  products  can  be 
designed  for  information  quality  variability  over  time  in  a  manner  somewhat  akin  to  system  reliability 
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growth  in  more  traditional  system  reliability  design  [3].  This  would  imply  that  processes  can  be 
formalized  and  inculcated  into  these  information  systems  that  achieve  and  maintain  varying 
appropriate  target  levels  of  agreed  upon  quality  criteria  well  past  the  point  of  release  to  the  end  user. 

In  fact,  it  is  natural  to  consider  that  there  would  be  a  subset  of  manufacturing  activities  that  would 
not  even  begin  until  a  considerable  time  after  the  initial  product  release  to  the  end  user  has  transpired. 
At  an  appropriate  time,  these  activities  would  create  or  engage  specific  sub-components  of  the 
information  product  to  provide  real  time  quality  target  level  achievement  as  required.  An  example  of 
this  concept  that  comes  to  mine  would  be  the  intentional  delay  of  a  real  time,  high  resolution  Preditor 
video  feed  until  a  predetermined  time  for  its  need  is  reached.  Providing  this  feed  earlier  than  needed 
might  distract  the  user  and  most  certainly  would  marginally  contribute  to  information  overload. 
Providing  this  feed  later  than  needed  by  the  user  degrades  both  the  information  quality  and  service 
reliability. 

As  an  example,  consider  the  Special  Operations  Force  (SOF)  planning  sequence  introduced  in  the 
Naval  Special  Warfare  Group- 1  case  study  [30]  shown  in  Figure  4.  The  overall  mission  sequence 
undergoes  a  series  of  distinct  activities  culminating  in  a  successful  mission  end  state.  Table  2 
illustrates  a  hypothetical  dynamic  scheduling  of  product  element  sequence  deployments  in  direct 
support  of  mission  planning  and  execution.  Table  3  then  shows  the  information  product  quality 
criteria  requirements  aligned  with  these  same  product  element  deployments. 

In  this  hypothetical  example,  the  levels  of  quality  criteria  are  specified  relative  to  a  SEAL  team’s 
identified  operational  needs  over  time.  Identifying  these  requirements  should  be  part  of  the  early 
stages  in  mission  planning,  well  in  advance  of  an  operation’s  start  point,  and  inculcated  into  the 
manufacturing  processes  in-place  that  create  the  specific  SEAL  team’s  information  products. 

A  highlight  noted  in  the  NSWG-1  case  study  underscores  the  need  for  manufacturing  processes  to 
be  adaptable  enough  to  be  able  to  effectively  respond  to  ad  hoc  requirements  outside  of  the  schedule 
as  well.  In  the  NSWG-1  case  study,  this  was  noted  as  the  ability  of  the  MSC  to  tailor  the  information 
products  to 
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Figure  4.  SOF  planning  sequence  extract  (source:  NSWG-1  Case  Study,  Booz  Allen  Hamilton). 


meet  the  evolving  needs  of  the  SEAL  teams.  Their  manufacturing  process,  largely  by  serendipity, 
was  capable  of  effectively  responding  to  changing  user  requirements  for  both  format  and  content.  In 
addition  to  what  was  noted  therein,  it  is  also  critically  important  to  sustain  the  purpose  of  the 
information  product  as  well:  to  support  accurate  inference  concerning  variable,  vague,  unknown,  and 
possibly  random  elements  of  the  battlespace  environment 

The  content,  functionality,  and  structure  are  subsequently  tailored  in  their  delivery  to  accommodate 
these  quality  requirements.  This  approach  would  then  be  applied  to  the  design  of  individual 
information  products,  or  classes  of  products  in  explicit  recognition  that  one  could  maximize  the 
effectiveness  of  an  information  product  by  aligning  its  evolution  with  that  of  the  environment  into 
which  it  is  to  reside.  This  is  a  transformation  concept,  and  an  example  of  designing  an  information 
product  for  quality  growth ,  a  natural  capability  of  dynamic  digital  information  products.  One  cannot 
help  but  recognize  a  need  for  systems  designers  to  assume  an  information  product  focus  in  order  to 
structure  this  type  of  dynamic  performance. 

In  comparison  to  six  standard  structured  systems  development  models  [45],  this  notion  is  quite 
distinct.  The  goal  of  a  manufacturing  process  is  to  insure  that  the  information  product  has  all  the 
required  levels  of  quality  at  precisely  the  time  they  are  needed.  We  simply  contend  that  this  should 
not  exclusively  be  enforced  at  the  point  of  release  to  the  end  user.  In  this  manner,  the  information 
product  maintains  “completeness”  throughout  its  effective  life  cycle  even  though  some  components 
are  present  at  lower  (and  higher)  levels  of  quality  criteria.  The  manufacturing  is  as  it  was  intended  to 
be  in  order  to  maximize  support  the  purpose  of  its  existence  in  the  first  place. 


Sequence 

Description 

Time-Point 
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*0 

Initial  deployment  of  information 

product  to  user 

h 

Mission  feasibility  assessment 

T2 

Support  plans 

h 

EXORD 

Engagement  ROE  assessment 

T5 

Objective  window  open 

T 6 

Objective  window  closed 

Mission  assessment 

Table  2.  Hypothetical  mission-critical  information  product  deployment  time  points. 


Design  Quality  Criteria 

^0 

T2 

h 

T4 

T5 

T6 

Completeness 

1.0 

1.0 

1.0 

1.0 

1.0 

1.0 

1.0 

1.0 

Conciseness 

1.0 

1.0 

1.0 

1.0 

1.0 

1.0 

1.0 

0.5 

Consistency 

0.8 

1.0 

0.8 

1.0 

1.0 

1.0 

0.7 

0.8 

Currency 

0.7 

0.9 

0.9 

0.9 

1.0 

1.0 

0.9 

1.0 

Traceability 

0.5 

1.0 

0.6 

0.9 

1.0 

0.9 

0.7 

0.8 

Comprehensiveness 

0.7 

1.0 

0.8 

0.9 

1.0 

0.9 

0.3 

1.0 

Accuracy 

0.7 

0.8 

0.9 

1.0 

1.0 

1.0 

0.7 

0.8 

Clarity 

0.8 

0.9 

0.6 

0.9 

1.0 

1.0 

0.8 

1.0 

Table  3.  Minimum  required  information  product  quality  criteria  levels  over  the  product’s  mission  life 

cycle. 


4.4  Life  Cycle  Stages 

In  general,  an  information  product  life  cycle  can  be  defined  as  the  stages  through  which 
information  passes  from  introduction  to  obsolescence  [29].  Based  on  the  concept  of  quality,  we  posit 
four  stages  characterized  by  the  levels  of  quality  present  in  the  product:  creation  (introduction), 
growth,  maturity,  and  decline.  The  creation  stage  culminates  with  the  levels  of  quality  present  upon 
release  to  the  end  user  ( r0 ). 

The  growth  stage  recognizes  the  dynamic  nature  of  information  products  as  described  earlier. 


19 


During  this  stage,  process  managers  focus  on  successfully  accomplishing  those  activities  designed  to 
raise  or  release  specific  levels  of  quality  criteria  in  accordance  with  a  life  cycle  schedule,  until  the 
quality  levels  reach  a  maximum  level  in  all  categories,  as  illustrated  in  Table  3  at  time  r4  . 

The  maturity  stage  immediately  follows  the  point  of  maximum  quality.  During  this  stage,  process 
managers  progressively  shift  the  focus  of  their  activities  to  maintain  alignment  with  the  schedule, 
allowing  non-critical  quality  levels  to  naturally  decline  or  sustain  without  intervention  while 
committing  resources  and  effort  to  raising  or  maintaining  high  levels  of  quality  in  critical  criteria. 

Finally,  the  decline  stage  is  as  it  suggests:  a  period  in  which  the  product  may  or  may  not  undergo 
re-use  or  renovation,  depending  upon  its  design.  Researchers  who  attempt  to  capture  the  life  cycle  of 
an  information  product  tend  to  focus  exclusively  on  creating  functional  descriptions  of  what 
transpires  concerning  some  dimension  of  measurement  on  information.  Most  assume  that  time  is  the 
dominant  factor  against  which  the  measure  of  information  declines.  This  may  or  may  not  be  the  case. 
All  assume  that  decline  is  inevitable  across  all  quality  criteria,  a  point  to  which  we  do  not  concur. 

The  decline  stage  is  a  period  in  which  process  managers  are  not  actively  attempting  to  adjust  the 
quality  levels  with  regards  to  the  product.  In  a  sense,  it  is  a  transient  period  for  the  information 
product  in  which  its  fundamental  quality  response  to  the  environment  within  which  it  finds  itself 
takes  place.  Some  levels  of  quality  are  unaffected,  some  deteriorate,  and  some  quite  possibly 
improve.  Exactly  which  criteria  respond,  and  in  what  manner,  depend  upon  the  particular  structure  of 
the  information  product  and  the  environmental  factors  present  that  are  able  to  affect  change. 

We  note  for  completeness  that  not  all  military  information  products  reach  the  decline  stage. 

Whether  they  do  or  do  not  is  a  function  of  the  mission  they  are  deployed  to  support.  In  the  NSWG-1 
case  study,  once  the  SEAL  teams  complete  a  specific  mission,  the  information  products  supporting 
that  mission  could  enter  a  decline  stage  on  the  assumption  that  the  team  would  be  shifting  its 
operations  to  address  new  objectives  and  the  old  products  are  no  longer  needed.  Some  may  never 
reach  the  decline  stage.  These  continue  to  mature,  entering  re-manufacturing  processes  that  adjust 
quality  criteria  levels  until  their  re-use.  Digital  imagery  information  products  are  a  good  example  of 
this. 

What  are  these  processes  that  need  to  be  managed?  Where  are  they  located  in  the  overall 
information  manufacturing  system?  When  should  they  be  performed?  What  affects  do  they 
introduce?  Is  there  a  priority  ordering  of  these  based  on  some  notion  of  marginal  returns  to  mission 
effectiveness?  Who  should  perform  them?  These  are  among  the  myriad  of  open  questions  that  arise  in 
this  context,  all  of  which  have  answers  that  contribute  significantly  to  understanding  the  quality  state 
of  an  information  product. 

With  this  in  mind,  we  proceed  to  investigate  salient  issues  pertaining  to  information  product 
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quality  in  this  study  with  the  intention  of  presenting  as  complete  a  framework  as  possible.  A  host  of 
issues  are  still  open  and  we  mention  these  where  they  occur.  These  will  require  further  investigation 
to  reach  conclusion. 

Not  surprisingly,  managers  in  knowledge  intensive  commercial  firms  have  come  to  realize  that 
“information  quality  is  a  decisive  competitive  factor  and  that  non-quality  information  can  be  a  crucial 
cost  driver”  [2].  For  the  military,  these  costs  tend  to  be  amplified  because  of  the  inclusion  of 
potential  loss  of  life  into  their  mix. 

5.  Information  Manufacturing  Representations 

DSIS  are  the  type  of  networks  of  interest  to  this  study,  ones  that  provide  a  critical  service  to  some 
element  of  military  force  units  that  must  make  decisions  affecting  missions  undertaken  on  the  part  of 
tactical  operations.  By  conceptualizing  the  various  activities  that  create,  format,  distribute,  and 
maintain  information  as  a  manufacturing  process,  it  is  possible  to  separate  concerns  over  the  quality 
of  these  products  from  the  quality  and  reliability  of  the  services  these  products  provide  to  the  users  of 
these  DSIS. 

Figure  5  provides  a  representation  of  DSIS  for  the  case  of  CTF-50  naval  forces  case  study  [53].  In 
it,  both  sensor  networks  and  human  transmission  serve  as  an  entry  point  for  resource  streams  of 
primitive  data  for  the  ensuing  manufacturing  processes.  For  judiciousness  in  exposition  and  without 
loss  of  generality,  we  aggregate  all  such  processes  at  a  single  point  of  manufacturing  in  the  diagram. 

In  practice,  it  is  more  the  case  that  various  intermediate  information  products  are  created  to  satisfy 
demands  of  users  throughout  the  system  in  addition  to  the  principal  target  for  the  final  information 
product(s).  These  intermediate  information  products  can  also  be  an  assemblage  of  many  elements  of 
primitive  data  and  intermediate  information  products. 

To  add  to  the  complexity  in  representing  such  systems,  there  also  exist  manufacturing  points  in  a 
DSIS  that  rely  on  human  cognition  merged  with  individual  knowledge  to  modify  an  existing 
intermediate  product  before  passing  it  on  through  the  system.  It  is  possible  to  classify  these 
manufacturing  points  as  being  objective  or  subjective  in  nature  depending  upon  whether  such  points 
incorporate  describable  processes  that  can  be  readily  automated,  or  complicated  analytical  processes 
that  further  complicate  modeling  efforts. 

At  an  Analysis  Facility,  individuals  or  software  applications  transform  data  from  the  resource 
streams  into  information  products  in  concert  with  predefined  rules.  These  rules  run  the  gamut  from 
simple  to  complex.  For  example,  a  simple  transformation  rule  could  define  an  order  in  which  data  of 
type  1  always  precedes  data  of  type  2.  Or,  the  transformation  rule  could  be  exceptionally  complex, 
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involving  for  example,  sequential  or  parallel  fusion,  image  processing  and  filtering  steps,  statistical 
analyses  of  results,  and  so  on. 

The  resulting  information  product  is  then  passed  to  a  Synthesis  Facility.  The  Synthesis  Facility 
assembles  the  various  information  product  components  into  a  single,  cogent  information  product. 


Figure  5.  Spatial  arrangement  for  components  of  information  manufacturing  in  CTF-50. 

A  truly  classic  example  of  a  Synthesis  Facility  is  the  Mission  Support  Center  (MSC)  in  San  Diego, 
California,  that  was  created  to  provide  reachback  support  for  NSWG-1  SEAL  teams  operating  in 
Afghanistan  and  Iraq.  When  intelligence  analysts  are  participants  in  this  manufacturing  process,  it  is 
quite  likely  that  the  information  components  will  be  integrated  with  organizational  experience, 
thereby  actually  creating  interpreted  information  we  call  knowledge.  Practically  speaking,  we  choose 
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to  not  explicitly  represent  a  knowledge  component  apart  from  an  information  component  because  it  is 
difficult  for  an  end  user  to  distinguish  between  the  two.  Whether  knowledge  components  can  be 
automatically  merged  into  information  products  remains  an  open  challenge  that  we  do  not  address. 
However,  the  storage  and  preservation  of  knowledge  for  any  organization  is  an  imperative  to  survival. 
Users  communicate  with  the  information  system  through  the  applications  interface  layer.  The  three 


Figure  6.  Information  product  manufacturing  for  NSWG-1  operations  in  OEF. 


applications  in  the  diagram  are  so  noted  because  they  arise  in  the  CTF-50  case  study  and  are  excellent 
examples  of  the  kinds  of  interfaces  that  NCO  systems  currently  exploit.  The  reliability  of  the 
transactions  that  take  place  across  this  interface  is  what  we  define  as  service  reliability. 

Figure  6  illustrates  how  the  same  concept  applies  to  the  NSWG-1  use  of  the  Mission  Support 
Center  (MSC)  during  Operation  Enduring  Freedom  [30].  The  three  major  service  applications  shown 
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are:  A3  (a  relational  database  of  tailored  intelligence  products),  WEBBE  (a  multi-point  instant 
messaging  tool  with  voice-over  IP  capabilities),  and  GBS  (a  broadband  satellite  downlink  that 
provided  for  fast  transfers  of  large  data  files).  The  MSC  is  a  Synthesis  Facility  comprised  of  Analysis 
Facilities  representing  each  of  the  seven  major  intelligence  and  information  resource  streams  being 
incorporated  into  information  products:  national  intelligence  agencies,  meteorology  &  oceanographic 
support,  service  specific  organizations,  consolidated 


Data  Source 
Block 


Input  source  for  raw  data  or  intermediate 
information  products. 


Processing 

Block 


Manipulates  or  combines  appropriate  data 
units  into  a  form  needed  for  manufacturing  the 
final  information  product(s). 


Data  Storage 
Block 


Represents  the  placement  of  data  units  or 
intermediate  information  products  in  files  or 
databases  where  they  are  available  to  the 
manufacturing  process  as  needed. 


Represents  activities  that  enhance  specific 
levels  of  information  quality  criteria  of 
information  products  to  meet  design 
specifications. 


Represents  the  final  information  product 
output  from  the  manufacturing  system. 


Figure  7.  Components  of  an  information  manufacturing  system. 


logistics,  theater  intelligence  agencies,  Department  of  State  embassies,  and  SOCOM/SOCJIC. 

The  special  arrangement  of  data  sources,  analysis  facilities,  and  synthesis  facilities  illustrated  in 
Figures  5  and  6  are  high  level  representations  of  an  information  product  manufacturing  system.  A 
more  detailed  representation  is  possible  as  well  using  a  flow  diagram  of  basic  system  components 
proposed  by  Ballou  et  al.  [31]  shown  in  Figure  7.  Once  these  blocks  are  linked  using  the  physical 
flow  characteristics  of  data  units  (DU)  or  information  units  (IU)  through  the  system,  a  low-level 
analysis  of  quality  can  proceed  as  the  product  progresses  towards  the  FIP  block. 

Figure  8  illustrates  a  spatial  arrangement  form  that  the  CIA  intelligence  resource  data  stream 
supporting  the  MSC  could  assume.  Within  the  diagram,  there  are  4  quality  blocks  (QB)  shown.  Data 
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originates  from  both  HUMINT  and  SIGINT  sources  in  a  particular  theater  of  operations.  QB1  and 
QB2  are  quality  blocks  designed  to  formally  review  and  verify  target  level  of  accuracy,  currency  and 
traceability  of  field  data  prior  to  it  being  transmitted  into  the  CIA  database.  These  are  examples  of 
feedback  controls  that  target  output  quantities,  adjusting  quality  levels  by  direct  interaction  with  the 
input  sources.  QB3  is  another  feedback  quality  block  in-place  to  maintain  the  integrity  of  the 


database,  insuring  the  completeness  and  currency  of 


the  data  records.  The  CIA  station  personnel  at  the  MSC,  along  with  their  specific  instrumentation  and 
computer  equipment,  comprise  the  CIA  MSC  Processing  block.  The  output  of  their  activities  is  an 
intermediate  information  product  (IIP)  that  becomes  part  of  an  integrated  intelligence  resource  stream 
used  in  the  manufacturing  of  the  MSC’s  information  products  delivered  to  the  end  user  through  the 
applications  layer.  QB4  is  a  feed  forward  quality  control  that  is  used  to  forecast  fluctuations  in 
critical  quality  criteria  levels  to  the  MSC  Synthesis  Facility  personnel.  This  diagram  consciously 
forces  the  CIA  Analysis  Facility  managers  to  decide  where  quality  blocks  should  be  located,  what 
type  of  control  should  the  impose  on  the  information  product  flow,  what  type  of  activities  should 
constitute  these  controls,  how  are  these  activities  to  be  performed  and  by  whom,  and  so  on. 

If  the  activities  that  act  on  the  information  product  in  a  processing  block  are  sufficiently  well- 
defined  and  simple,  then  quantitatively  tracking  changes  in  quality  criteria  levels  could  be 
accomplished  via  a  straightforward  calculus.  For  example,  suppose  that  y  represented  the  processing 
block’s  output  value  of  currency  of  the  information  product,  and  it  is  the  result  of  a  comparison 
between  the  differences  of  two  inputs:  xx  and  x2  at  any  time  t.  The  quality  effect  of  this  processing 
block  on  currency  would  then  be  fq  (jq ,  x2 ,  t)  =  xx  (t)  -  x2  (t) .  However,  when  processing  is  complex, 
it  may  be  necessary  to  substitute  subjectively  derived  quality  response  functions  for  a  calculus-based 
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analysis  in  order  to  track  quality  criteria  changes.  The  issue  of  exactly  how  to  measure  the  quality 
attributes  as  they  relate  to  information  products  is  still  an  open  question. 

During  the  design  phase  for  an  information  manufacturing  process  the  construction  of  the  flow 
chart  itself  has  value  independent  of  any  subsequent  analysis.  It  forces  an  NCO  organization  to 
consciously  position  quality  blocks  in  the  flow,  thereby  at  least  illuminating  high  priority  points  in  the 
production  sequence  where  quality  should  be  insured  in  some  manner.  These  quality  blocks  provide 
controls  necessary  for  information  quality  maintenance  planning.  Whether  these  controls  are 
implemented  using  automated  procedures,  software  applications,  managerial  intervention,  or  some 
other  means  is  largely  driven  by  the  purpose  of  the  information  product  and  the  nature  of  the 
organization  within  which  manufacturing  is  going  to  take  place. 

6.  Impact  on  Existing  Theory 

An  information  product  manufacturing  framework  is  extremely  valuable  to  proper  design  and 
construction  of  high  quality  information  products  and  the  processes  that  form  these  products  for  use. 
At  this  point,  however,  it  is  important  to  note  that  there  are  no  existing  quantitative  characterizations 
of  quality  criteria  capable  of  being  directly  applied  to  this  framework.  Even  those  specifically  created 
to  facilitate  discussion  of  quality  criteria  in  the  context  of  information  superiority  [23]  are  inadequate 
for  the  task,  principally  because  their  adopted  framework  and  its  relative  context  differ  significantly 
from  the  framework  developed  in  this  study.  The  conditional  representation  that  lies  at  the  heart  of 
their  modeling  approach  is  nearly  identical  to  that  used  to  capture  uncertainty  flows  in  sensor-to- 
shooter  systems  [7]  whose  focus  is  on  elements  of  reliability  as  opposed  to  quality. 

High  quality  information  makes  it  easier  to  transform  information  into  knowledge  by  helping  to 
interpret  and  evaluate  the  information,  by  assisting  in  connecting  to  prior  knowledge,  and  by 
facilitating  the  application  of  the  information  in  new  contexts. 

One  method  of  measuring  the  levels  of  information  quality  on  a  more  aggregate  level  would  be  to 
assess  the  likelihood  that  the  information  would  get  turned  into  knowledge  a  priori ,  or  to  assess  the 
percentage  of  information  that  does  get  turned  into  knowledge  a  posteriori .  Assuming  that  the 
assemblage  of  an  information  product  is  guided  by  a  single  purpose  originating  from  a  single  intent 
enables  one  to  estimate  these  measures  for  each  of  the  quality  criteria  by  assessing  the  degree  to 
which  they  correspond  to  that  purpose.  This  is  one  promising  avenue  being  investigated  by  the 
authors. 

This  manufacturing  framework  agrees  with  that  of  the  NCOCF  in  that  one  must  implicitly  assume  it 
is  possible  to  describe  the  quality  of  information  using  a  finite  set  of  criteria  grouped  into  meaningful 
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dimensions  [2].  It  diverges  in  the  sense  that  it  defines  these  criteria  relative  to  a  specific  product. 
Survey  instruments  of  the  type  used  in  support  of  the  NCO  case  studies  mentioned  in  this  study  and 
other  recent  analyses  of  NCO  systems  are  recognized  as  the  principle  means  of  assessing  the  degree 
of  adherence  to  such  criteria  when  information  is  conceptualized  in  a  general  sense.  Thus,  while  we 
advocate  surveying  end  users  to  assess  the  subjective  criteria  related  to  service  reliability,  we  consider 
this  approach  inappropriate  for  assessing  the  quality  criteria  levels  of  information  products. 


Quality  of  Organic  and 
Individual  Information 

Objective  Fitness  for  Use 


Figure  9.  Eight  quality  criteria  currently  contained  in  NCOCF. 


The  eight  criteria  currently  specified  in  the  NCOCF  are  shown  in  Figure  9.  Perry  et  al.  [23]  define 
each  of  the  criteria  as  follows: 

•  Correctness  -  the  degree  to  which  information  agrees  with  ground  truth. 

•  Consistency  -  the  extent  to  which  information  is  in  agreement  with  related  or  prior 
information 

•  Currency  -  the  situation  independent  time  for  the  C4ISR  to  produce  and  distribute  a  CROP 

•  Precision  -  the  level  of  measurement  detail  in  an  information  item 

•  Completeness  -  the  degree  to  which  information  is  free  of  gaps 

•  Accuracy  -  the  appropriateness  of  the  precision  of  information  to  a  particular  use 

•  Relevance  -  the  proportion  of  information  collected  that  is  related  to  the  task  at  hand 

•  Timeliness  -  the  situation-dependent  degree  to  which  information  is  available  when  needed 
The  same  study  posited  the  belief  that  information  quality,  the  inherent  ‘goodness’  of  information, 

is  situation  independent.  This  is  an  existential  position  consistent  with  viewing  information  in  a 
general  sense.  We  agree  that  information  should  be  conceptualized  as  an  entity  apart  from  the 
technology  that  carries  it,  but  are  not  willing  to  assign  complete  situational  independence  to  its 
descriptive  quality  criteria,  especially  in  the  context  of  information  product  manufacturing. 
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While  an  information  product  can  assume  a  vast  array  of  forms,  underlying  all  forms  is  a  specific 
purpose  that  recognizes  the  information  product  is  to  be  used.  The  arrangement  of  content  and 
functions  associated  with  the  product  (its  design)  is  driven  by  this  purpose.  Purpose  is  a 
manifestation  of  intent  and  the  information  product,  if  properly  manufactured,  by  extension  is  a 
physical  manifestation  of  this  same  intent.  Part  of  the  process  of  validating  or  assessing  the 
information  quality,  and  reliability  as  we  explain  later,  lies  in  an  ability  to  start  with  the  final 
information  product  and  perform  inference  as  to  the  actual  intent  given  the 


Figure  10.  Layers  of  quality.  (Iverson:  Meta-Information  Quality ,  ICIQ  2001) 

evidence  present  in  the  product.  This  is  a  form  of  reverse  engineering  applied  to  an  abstract  notion  of 
product. 

In  the  commercial  sector,  information  products  in  advertising  are  manufactured  for  the  purpose  of 
shaping  consumer  perception  or  to  establish  distinctive  characteristics  that  set  apart,  or  strategically 
“place,”  a  product  within  its  competitive  landscape.  Whether  or  not  the  product  is  actually  purchased 
by  a  consumer  is  outside  of  the  control  of  the  information  product  manufacturers. 

In  a  military  context,  a  vast  majority  of  information  products  are  manufactured  with  the  purpose  of 
helping  decision-makers  perform  accurate  inference  estimates  concerning  critical  parameters  of 
battlespace  behavior.  This  purpose  establishes  relevancy  against  which  quality  characteristics  can  be 
meaningfully  defined.  When  viewed  through  the  lens  of  manufacturing  information  products,  these 
criteria  are  most  definitely  situation  dependent.  Such  a  distinction  would  not  be  evident  were  it  not 
for  this  approach. 

Both  this  new  perspective  and  the  existing  literature  bears  out  that  an  amount  of  revision  in  the 
stated  NCOCF  criteria  is  required.  Principally  this  is  so  because  several  of  the  current  quality  criteria 
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are  too  general  to  be  of  practical  use  in  the  design  of  information  products. 

Fundamentally,  issues  of  quality  exist  on  the  four  layers  shown  in  Figure  10.  The  objective  criteria 
of  precision  and  correctness  are  recognized  data  quality  issues  that  are  principally  the  focus  of 
information  assurance  efforts.  For  military  operations,  such  as  those  documented  in  the  NSWG-1 
case  study,  precision  translates  into  a  user  need  to  have  sufficiently  specified  location  data  to  facilitate 
operational  targeting.  This  requirement  is  arguably  a  topical  subset  of  accuracy,  as  explained  in  what 
follows. 

The  criterion  of  timeliness  is  actually  an  aggregate  measure  that  bears  further  decomposition  as  is 
explained  shortly.  The  criteria  of  relevance  is  generally  unworkable  as  a  criteria  for  information 
quality  because  it  contains  various  types  of  pertinence  that  should  be  distinguished,  not  all  of  which 
are  necessary  to  examine  in  the  context  of  military  NCO  systems.  Saraceciv  [10]  detailed  five 
principle  notions  of  relevance  as  they  apply  to  information  systems:  system,  topical,  cognitive, 
situational,  and  motivational. 

System  relevance  expresses  the  relation  between  a  query  and  the  resulting  information  objects. 
Direct  assessment  of  system  relevance  is  left  to  the  end  user  and  is  based  on  the  effectiveness  of  the 
received  information  objects  for  performing  inference. 

Topical  relevance,  which  expresses  the  degree  of  “aboutness”  contained  in  the  information  object, 
would  appear  to  be  automatic  for  most  DSIN  used  by  the  military  because  systems  exist  for  focused, 
mission-oriented  functionality.  Queries  in  this  setting  are  topically  relevant  or  they  do  not  exist. 

Cognitive  relevance  expresses  the  relation  between  the  user’s  state  of  knowledge  and  the  degree  to 
which  the  information  object  fulfills  the  cognitive  information  need  of  that  user.  It  follows  that  the 
degree  to  which  the  information  informs  the  user  appears  to  be  a  useful  direct  measurement  of  this 
criteria.  Notice  the  user-centric  nature  for  the  assessment  of  this  criterion. 

Situational  relevance  expresses  the  relation  between  the  situation,  task  or  problem  at-hand  and  the 
information  retrieved  from  the  system.  One  will  note  that  this  criterion  is  very  closely  aligned  with 
topical  relevance.  The  distinction  between  the  two  lies  in  its  intent:  this  notion  of  relevance  goes 
directly  to  the  usefulness  of  the  information  object  for  decision  making  and  reducing  uncertainty 
related  to  the  situation  at-hand.  In  this  sense,  situational  relevance  is  likely  to  be  what  was  intended 
in  the  specification  of  the  NCOCF  criteria  shown  in  Figure  2. 

Lastly,  and  perhaps  the  most  subjective  of  all  five  notions  of  relevance,  there  is  motivational 
relevance.  This  notion  of  relevance  expresses  the  relation  between  the  intents,  goals,  and  motivations 
of  the  user  and  the  information  retrieved  from  the  system.  The  most  common  aspects  of  measurement 
in  this  regard  are  user  satisfaction,  user  expressed  success  and  the  sense  of  accomplishment  afforded 
or  facilitated  by  the  information  object  provided  by  the  system. 
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A  note  on  trustworthiness  is  also  appropriate  here.  Although  it  is  not  contained  in  the  current 
NCOCF  criteria,  it  has  been  bantered  about  in  discussion  as  one  valuable  measure  of  quality.  As  a 
subjective  notion,  it  is  interesting  to  consider,  but  as  an  objective  criteria  it  is  too  high  of  a  subjective 
abstraction  to  be  of  worth  to  systems  designers.  Trust  of  information  products  is  attained  by  insuring 
that  the  appropriate  levels  of  core  objective  quality  criteria  are  met. 

In  all,  one  gets  the  sense  of  the  complexity  associated  with  including  relevance  as  one  criteria  of 
information  quality.  The  two  notions  of  relevance  that  readily  apply  to  military  systems,  cognitive 
and  situational,  both  rely  on  user-based  subjective  assessments.  It  is  for  this  reason  that  they  should 
more 
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Figure  11.  Information  product  manufacturing  quality  criteria  for  the  NCOCF. 


appropriately  appear  as  supporting  criteria  for  service  reliability. 

Of  the  16  standard  information  quality  criteria  accepted  in  practice  [2],  we  consider  ten  to  be 
candidates  for  defining  the  principle  quality  criteria  of  information  for  NCO  information  systems. 

The  criterion  known  as  correctness  (Is  the  information  free  of  distortion,  bias,  or  error?)  is  properly 
a  criterion  of  concern  for  information  assurance,  as  is  security  (Is  the  information  protected  against 
loss  or  corruption?).  We  also  exempt  interactivity  (Can  the  information  process  be  adapted  by  the 
end  user?),  accessibility  (Is  there  a  continuous  and  unobstructed  way  to  get  to  the  information?),  and 
speed  (Can  the  infrastructure  match  the  user’s  working  pace?),  and  convenience  to  be  elements  of 
service  reliability,  a  complementary  study  to  this  effort. 

The  criterion  known  as  timeliness  (Is  the  information  processed  and  delivered  rapidly  without 
delays?)  is  best  divided  into  two  components:  currency ,  which  is  part  of  the  criteria  we  already 
address,  and  volatility  (How  quickly  does  the  “risk-to-use”  change?  Aka:  shelf-life  of  risk.).  Because 
this  second  component  is  a  direct  measure  of  risk-to-use,  which  is  the  basis  of  our  reliability 
construct,  we  consider  it  as  part  of  reliability  as  opposed  to  quality. 
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Within  an  information  product  perspective,  the  nine  remaining  criteria  shown  in  Figure  1 1 
represent  the  span  of  quality  characteristics  appropriate  for  inclusion  in  some  manner  in  the  NCOCF. 
These  criteria  encompass  elements  of  both  dynamic  and  static  information  products. 

■  Comprehensiveness:  is  the  scope  of  the  information  product  adequate  for  its  intended 
purpose? 

o  Example:  weather  data  for  the  purpose  of  determining  the  feasibility  of  attacking  an 
objective  at  a  specific  date/time  is  bounded  +/-  three  hours, 
o  Counter-example:  weather  data  includes  areas,  dates,  and  times  not  relevant  to  the 
mission. 

■  Accuracy:  are  the  information  product  components  precise  enough  or  close  enough  to  reality 
for  its  intended  purpose? 

o  Example:  digital  imagery  in  support  of  targeting  ROE  determination  has  enough 
resolution  to  read  appropriate  religious  symbols  on  the  structure, 
o  Counter-example:  resolution  insufficient  to  distinguish  markings. 

■  Clarity:  is  the  information  product  understandable  to  the  end  user  in  each  component 
presentation  form  employed  during  its  lifecycle? 

o  Example:  critical  logistic  supply  levels  provided  to  the  end  user  in  graphical  display 
that  indicates  warning  levels  at  which  the  end  user  should  take  action, 
o  Counter-example:  critical  logistic  supply  levels  provided  to  the  end  user  in 
extensive  data  tables. 

■  Applicability:  can  the  information  product  be  directly  applied  by  the  end  user  at  its  sequence 
deployment  point  or  does  it  have  to  be  further  transformed  by  the  user? 

o  Example:  target  ground  location  for  the  purpose  of  scheduling  indirect  fires  is 
transmitted  to  SEAL  team  via  a  hand-held  device  using  GPS  data  within  1  meter, 
o  Counter-example:  target  ground  location  is  provided  using  UAV  digital  imagery 
that  must  be  fused  with  map  information  by  the  user. 

■  Conciseness:  is  the  information  product  to-the-point  and  devoid  of  unnecessary  or 
distracting  elements? 

o  Example:  situation  report  presented  in  bullet  list  or  aggregated  image  form, 
o  Counter-example:  situation  report  presented  in  prose  form  with  in-depth 
explanations  of  standard  terms. 

■  Consistency:  is  the  information  product  free  of  contradictions  and  convention  breaks  at  any 
point  in  its  lifecycle? 
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o  Example:  each  open  window  of  a  multiple  window  display  used  for  estimating 
enemy  situational  awareness  provides  complementary  information  in  a  form 
previously  used  by  the  end  user  during  training, 
o  Counter-example:  friendly  situational  awareness  information  presented  in  an 
unfamiliar  format  (e.g.,  UK  Force  use  of  FCBC2  in  Iraq.). 

■  Currency:  is  the  information  up-to-date  relative  to  its  resource  at  each  sequence  deployment 
point  in  its  lifecycle? 

o  Example:  SEAL  teams  displayed  intelligence  information  exactly  matches  that  in 
the  DIA  database  at  the  MSC  facility. 

o  Counter-example:  system  limitations  allow  DIA  intelligence  updates  to  SEAL 
teams  to  occur  on-the-hour. 

■  Completeness:  are  the  product  components  and  their  functional  elements  present  in 
accordance  with  the  time  sequencing  of  delivery  to  the  end  user? 

o  Example:  high  resolution  digital  video  feed  not  present  during  initial  product  release 
(low  resolution  at  that  point)  but  product  is  preset  to  begin  streaming  on-demand  by 
end  user  when  ROE  determination  will  need  to  be  made, 
o  Counter-example:  product  incapable  of  switching  to  high  resolution  digital  video 
feed  when  needed  without  complete  re-manufacturing  of  the  information  product. 

■  Traceability:  are  the  sources  supporting  the  information  product  visible  to  the  end  user  at 
points  where  this  is  needed? 

o  Example:  end  user  is  capable  of  quickly  determining  the  source  of  contradictory 
intelligence  information  for  use  in  a  ROE  determination, 
o  Counter-example:  end  user  cannot  resolve  contradictory  intelligence  information 
back  to  origination  source. 

Notice  that  completeness  is  defined  as  the  degree  to  which  the  information  product  content  contains 
the  intended  manufacturing  components  and  functions  supporting  its  purpose.  This  definition  moves 
the  assessment  metric  away  from  a  comparison  with  ground  truth,  which  continuously  evolves  and  is 
never  assessable  beyond  estimation  with  uncertainty,  to  a  tangible  product  whose  design  and  content 
are  under  the  control  of  the  manufacturing  system. 

The  last  criterion  is  new  to  the  NCOCF.  Information  should  be  traceable  at  some  level  by 
information  managers.  This  concern  arose  in  the  CTF-50  case  study  as  a  critical,  yet  easily 
overlooked  criterion  needed  to  assess  the  quality  of  information  provided  by  the  Knowledge  Web 
(KWeb). 

One  of  the  reported  strengths  of  the  KWeb  application  was  the  fact  that  many  individual  users  had 
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the  ability  to  post  information  without  extensive  centralized  review.  The  system  exchange  of 
information  evolved  to  contain  a  strong  degree  of  self-correctiveness  because  information  was 
traceable  in  two  directions:  from  the  source  to  the  product  and  from  the  product  to  the  source.  This 
feature  enabled  users  high  in  the  command  structure  and  low  in  the  command  structure  to  validate 
posted  information  as  needed  in  real  time.  They  had  an  option  to  put  all  information  through  an 
approval  process  prior  to  posting,  but  opted  for  this  approach  instead. 


This  is  a  lucid  example  of  users  evolving  to  take  advantage  of  a  natural  feedback  cycle  in  the 
system  to  perform  the  information  maintenance  task  of  validating  postings  rather  than  creating  a 
separate  organization 


Figure  12.  Major  control  elements  for  manufacturing  information  process. 


to  perform  this  task  for  the  entire  system.  By  leveraging  both  the  system  capability  and  the 
motivation  among  the  user  pool  to  be  working  with  the  highest  quality  information  possible,  the 
KWeb  evolved  to  have  a  feature  that  went  a  long  way  towards  establishing  both  trust  and  confidence 
in  the  user  base. 

7.  Information  Product  Maintainability 

Fundamental  to  the  concepts  of  designing  and  manufacturing  high  quality  and  highly  reliable 
information  products  is  the  associated  belief  that  these  valuable  product  characteristics  can  be 
maintained.  Traditionally,  one  would  conceptualize  internal  and  external  control  features  [24]  to 
accomplish  this.  For  military  information  product  manufacturing  for  decision  support,  distinguishing 
between  user  processes  and  manufacturing  system  processes  is  more  helpful  for  system  designers. 

The  end  user  reliance  on  social  network  structures  in  the  NSWG-1  case  study  as  an  adaptive  work¬ 
around  created  to  overcome  some  deficiency  of  the  information  product  is  an  example  of  a  user 
control  process.  A  formalized  capability  of  the  MSC  programmers  to  create  tailored  information 
products  for  the  NSWG  teams  on  demand  is  an  example  of  a  process  intentionally  imbedded  into  the 
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manufacturing  process  to  maintain  high  quality. 

Maintaining  high  levels  of  quality  and  reliability  has  in  the  past  essentially  meant  building 
appropriate  controls  exclusively  into  the  manufacturing  process.  In  an  earlier  example,  four  quality 
blocks  (QB)  were  inserted  into  the  CIA  Analysis  Facility  that  focused  solely  on  maintaining  a  high 
quality  information  product.  To  be  complete,  Figure  8  should  also  show  the  particular  structural 
controls  on  the  CIA  MSC  Processing  activity  itself  that  would  target  maintaining  a  high  quality 
process  as  well. 

It  was  this  piece  of  the  product  life  cycle  that  represented  the  dynamic  element  of  the 
manufacturing  chain  because  early  information  products  tended  to  be  predominantly  static. 
Information  products  such  as  printed  reports  and  books  still  align  with  this  notion.  When  this  type  of 
product  is  released  to  an  end  user,  the  “Little-q”  and  “Little-r”  instantiated  levels  of  quality  and 
reliability  are  assessed.  Assuming  that  these  levels  are  acceptable,  the  product  is  released  to  the 
consumer.  Past  this  point,  quality  and  reliability  are  a  concern  to  the  original  manufacturer  only  so 
far  as  warranty  obligations  and  re-manufacturing  opportunities  that  extended  the  value-chain  for  these 
manufactured  products  are  concerned. 

Digital  information  products  manufactured  in  support  of  military  decision-making  in  an  NCO 
environment  however,  are  quite  different.  They  quite  likely  have  dynamic  elements  that  possess 
imbedded  functions  directly  connected  to  or  dependent  upon  specific  portions  of  the  manufacturing 
process.  Hence,  it  is  useful  to  consider  the  maintainability  of  information  products  both  prior  to  the 
initial  release  to  the  end  user  and  during  its  life  cycle. 

There  are  three  modes  of  control  are  appropriate  to  consider  when  addressing  maintenance  of 
information  products:  structural  control,  feed  forward  control,  and  feedback  control.  The  positioning 
of  these  control  elements  in  the  flow  of  process  is  shown  in  Figure  12  and  applied  earlier  in  Figure  8. 

Structural  control  is  internal  to  the  information  product  itself  made  possible  by  the  interaction  of  a 
subset  of  the  dynamic  components  of  the  product.  This  is  a  unique  definition  quite  different  from  the 
traditional  notion  of  structural  controls,  which  are  thought  of  a  process  controls.  These  dynamic 
components  maintain  an  umbilical  link  to  resource  flows  that  are  not  severed  upon  deployment  of  the 
product  to  the  user. 

Structural  controls  do  not  rely  upon  regular  monitoring  of  inputs  and  outputs.  For  the  NSWF-1 
case  study,  MSC  personnel  shift  schedules,  recruitment  and  assignment  criteria  for  assigning 
personnel  to  the  MSC,  rotational  assignments  from  CONUS  MSC  locations  to  in-theater  operation 
sites,  are  examples  of  structural  controls.  In  a  more  casual  sense,  these  controls  define  the  activities 
that  are  once  (and  more)  removed  from  touching  the  manufacturing  process,  but  are  critical  to 
maintaining  a  high  quality  or  highly  reliable  information  product. 
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Feed  forward  control  relies  upon  regular  monitoring  of  inputs  to  predict  future  variation  in  product 
quality  or  reliability  in  order  that  human  intervention  or  triggered  automated  system  action  may  be 
taken  to  prevent  such  degradation.  An  example  of  a  feed  forward  control  for  COP  quality  would  be 
the  continuous  monitoring  of  Dragon  Eye  UAV  operational  status  (e.g.,  condition,  fuel,  camera 
status,  etc.)  that  is  providing  a  live  digital  imagery  of  a  target.  If  the  fuel  levels  of  the  on-site  Dragon 
Eye  UAV  was  anticipated  to  drop  to  the  point  that  it  could  no  longer  logger  onsite  to  support  its 
information  feed,  a  supplemental  Dragon  Eye  UAV  could  be  launched  to  insure  that  the  COP  quality 
in  this  dimension  did  not  drop  below  acceptable  levels  during  a  critical  time  window.  Feed  forward 
control  defines  pro-activity  with  regards  to  information  product  manufacturing.  Systems 
incorporating  feed  forward  and/or  structural  controls  only  are  referred  to  as  open-loop  systems. 

Feedback  control  relies  upon  regular  monitoring  of  outputs  to  assess  deviations  in  product  quality 
or  reliability  which  indicate  the  need  for  corrective  action  to  restore  or  adjust  the  product  to  its 
desired  state.  Systems  incorporating  this  form  of  control  are  referred  to  as  closed-loop  systems. 

Our  interest  in  maintainability  with  respect  to  information  products  is  several-fold.  First, 
constructing  a  framework  for  identifying  information  quality  maintenance  locations  in  the  flow  of 
intermediate  information  products  within  the  manufacturing  process  would  enable  systems  designers 
to  imbed  product  and  process  oversight  that  would  contribute  to  the  requirement  of  adaptability 
identified  earlier. 

Second,  it  would  enable  information  managers  to  have  focused  activities  at  different  stages  of  the 
product’s  life  cycle.  In  addition  to  creating  efficiency  with  respect  to  effort,  this  would  enable 
managers  to  identify  the  critical  skills  needed  by  maintenance  personnel,  thereby  contributing  to 
structural  controls  necessary  to  insure  the  long  term  health  of  the  manufacturing  facility  itself. 

Ballou’s  concept  of  positioning  quality  blocks  does  not  include  feedback  or  feed  forward  controls. 
He  envisions  that  blocks  would  be  positioned  to  interrupt  the  flow  of  information,  fix  the  deficiency, 
and  allow  the  product  to  proceed  only  upon  restoration  or  achievement  of  an  acceptable  Little-q 
quality  threshold.  While  acceptable  for  static  information  products  that  do  not  evolve  their  quality 
criteria  as  we  propose,  dynamic  products  require  dynamic  feed  controls  in-place  to  properly  maintain 
information  product  quality.  These  controls  can  be  positioned  in  the  manufacturing  process  to 
augment  rather  than  interrupt  the  flow  of  information  products  to  an  end  user. 

7. 1  The  Information  Sharing  NCW  Tenet ,  NSWG-1  Case  Study 

Information  sharing  in  general  does  not  necessarily  enhance  the  quality  of  information  products.  In 
the  manufacturing  framework,  quality  is  an  intrinsic  characteristic  of  an  information  product  that  is 
intentionally  designed  and  controlled  throughout  its  life  cycle.  It  is  directly  dependent  upon  the 
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individual  components  comprising  the  product,  their  collective  representation,  and  the  functions  they 
perform  for  the  end  user  over  time.  Information  sharing  is  one  part  of  a  process  that  can  lead  to 
improved  information  quality,  but  only  in  a  system  that  has  carefully  designed  and  deployed  product 
manufacturing  processes  with  empowered  information  quality  managers  having  access  and  authority 
to  revise,  update,  refresh,  repair,  or  replace  components  of  the  information  product.  Otherwise  it  is  no 
more  than  the  fortune  alignment  of  people  and  activities  touching  the  product  through  information 
sharing,  the  end  result  being  an  observed  increase  in  product  quality. 

The  NCW  tenet  concerning  information  sharing  tacitly  assumes  these  critical  requirements  are 
present  in  successful  system  implementations.  We  conjecture  that  the  reason  it  is  not  explicitly  stated 
as  such  is  that  the  generic  concept  of  information  currently  held  is  insufficient  to  identify  this 
shortcoming  without  the  concept  of  information  product  manufacturing.  For  the  NSWG-1  case  study, 
observers  surmised  that 
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Figure  13.  Observed  NCW  effects  on  information  quality.  (Source:  Booz  Allen  Hamilton  [30]). 


“improvements  in  the  degree  of  information  shareability  led  to  increases  in  information  quality” 
(Figure  13). 

That  the  “quality  of  information  increased”  during  the  MSC  transition  period  from  OEF  to  OIF  is 
by  now  clearly  too  abstract  of  an  observation  to  be  actionable.  That  it  directly  followed  from 
improving  the  ability  to  share  information,  which  is  almost  exclusively  a  hardware  and  software 
issue,  is  doubtful.  More  likely  is  the  case  that  many  of  the  improvements  and  enhancements  in  the 
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information  product  manufacturing  system  were  in  fact  observed  but  misdiagnosed.  Again,  we 
believe  that  had  this  information  manufacturing  framework  been  adopted  and  in-place  prior  to  the 
case  study  and  the  language  of  manufacturing  been  available  to  them,  then  the  actual  story  would 
have  read  more  as  shown  in  Figure  14. 

Information  sharing  occurring  via  a  COP  in  support  of  decision-making  absolutely  enhances 
operational  collaboration  across  dissimilar  units,  as  long  as  the  COP  was  properly  designed, 
manufactured  and  reliably  delivered  through  the  applications  interface  to  the  end  user.  This  is  one  of 
the  synergistic  effects  NCO  should  foster  and  develop.  Collaboration  is  an  active  process  in  which 
parties  involved  commit  to  align  effort  under  a  unified  purpose  or  intent.  This  unified  purpose  or 
intent  is  consistent  with  what  Perry  et  al.  [23]  refer  to  as  establishing  common  ground  among 
collaborators,  a  critical  determinant  of  success. 

Information  sharing  in  support  of  collaboration  assumes  that  competent  rational  decision  makers 
will  exploit  such  information  products  when  they  receive  them  to  the  best  of  their  abilities  within  the 
limits  of  the  resources  they  control.  When  this  occurs,  collaboration  is  a  success.  When  it  does  not 
happen,  collaboration  has  failed.  Information  sharing  can  take  place  regardless  of  the  end  state. 
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Figure  14.  Observed  NSWG-1  NCO  effects  from  information  manufacturing  perspective. 


With  regards  to  the  NSWG-1  case  study,  there  are  a  host  of  observations  and  questions  arising  from 
adopting  an  information  product  manufacturing  perspective  that  remain  unaddressed.  A  select  number 


37 


of  them  are  briefly  listed  in  what  follows. 

■  “Leveraging  reachback  support  at  the  MSC  was  a  source  of  success.” 

o  What  specifically  happened  to  the  information  products  being  provided  to  the 
SEAL  teams? 

■  “70  -  75%  reduction  in  the  forward  footprint” 

o  What  specific  personnel  and  units  were  shifted? 

o  What  skill  sets  did  this  represent  in  the  context  of  manufacturing? 

■  “The  MSC  was  able  to  gather  all  information,  tailor  it  to  the  fleet. . .” 

o  The  system  had  flexible  manufacturing  points  that  could  rapidly  adapt  the  form  and 
content  of  the  information  product  to  the  evolving  end  user’s  needs  rather  than 
imposing  this  requirement  on  the  user  themselves  to  perform.  That  is,  the  end  user 
did  not  have  to  develop  “work-arounds”  (see  earlier  diagnostic). 

■  “The  MSC  was  a  conduit. . .” 

o  The  MSC  was  a  Synthesis  Facility  within  the  information  manufacturing  system. 

■  “Increased  quality  information  available  earlier  in  the  planning  process. . .” 

o  This  is  actually  a  service  reliability  issue  as  opposed  to  an  information  product 
quality  concern.  However,  the  observation  underscores  the  need  to  align  a 
schedule  of  Little-q  quality  level  product  deliveries  to  the  end  user  according  to 
their  needs. 

■  “The  hallmark  of  SOF  is  that  they  are  always  open  to  change  and  out-of-the-box 
thinking...” 

o  The  observation  pertains  to  user  characteristics  related  to  adaptability  but  does  not 
necessarily  relate  to  the  structure  and  content  of  information  products  they  require. 

■  “In  theater  personnel  with  operational  knowledge  of  the  MSC. . .” 

o  This  exactly  parallels  the  benefits  of  having  organizational  managers  who  have 
progressed  in  their  careers  having  spent  quality  time  as  part  of  the  manufacturing 
labor  force.  So  long  as  the  major  structural  organization  and  processes  of  the 
facility  remain  reasonably  consistent  with  their  experience,  they  can  effectively 
translate  their  knowledge  of  supply  resources,  manufacturing  capability,  and  best 
practices  of  process  execution  into  an  advantage  elsewhere  in  the  organization. 
They  are  also  imbued  with  a  cache  of  influence  recognized  by  those  remaining  in 
the  manufacturing  facility.  However,  had  the  organization  at  the  MSC  changed, 
the  resident  skill  sets  changed,  the  manufacturing  processes  changed  during  the 
period  from  OEF  to  OIF,  the  likelihood  that  this  observation  would  have  enhanced 
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NCO  would  be  extremely  small. 

■  “During  OEF,  the  MSC  command  relationships  were  ad  hoc  and  could  not  generate  RFI’s 
to  the  intelligence  community. . .” 

o  Formalized  supplier  relationships  did  not  exist,  leaving  the  MSC  to  compete  for 
supply  as  a  “walk  in”  customer.  Sporadic  supply  availability  translates  directly  to 
uncertain  product  availability  to  the  end  user.  Service  reliability  suffers 
dramatically  (and  nonlinearly),  decreasing  the  confidence  in  the  end  user  that  the 
product  will  be  present  when  needed.  In  this  situation,  strong  hierarchical 
organizations  such  as  military  units  can  at  best  impose  a  cautious  dependency  on 
the  part  of  the  end  user.  This  is  an  important  distinction  to  understand:  the  risk-to- 
use  has  not  changed  (information  reliability),  the  quality  of  the  product  has  not 
changed  (information  quality),  but  the  ability  of  the  system  to  furnish  the 
product  to  the  end  user  has  (service  reliability).  This  precisely  illustrates  the 
usefulness  of  decomposing  an  NCO  system  into  three  major  focus  areas  as  we  have 
done. 

On  the  1999  mission  capabilities  to  support  new  objectives: 

■  “Collocate  intelligence  and  operations  personnel  that  understood  the  intrinsic  and 
idiosyncratic  needs  of  NSW  operators  in  any  of  their  various  missions,  and  who  could 
provide  the  kind  of  specific  information  that  the  Joint  Intelligence  Centers  provided.” 

o  Build  an  adaptable,  capable  information  manufacturing  process  with  high  service 
reliability  to  the  end  user. 

■  “Develop  liaisons  with  national  intelligence  agencies.” 

o  Establish  formal  links  to  resource  suppliers  relevant  to  the  specific  information 
products  being  manufactured. 

■  “Centralize  blue-force  monitoring  via  “black  boxes”  that  deployed  NSW  forces  would 
carry,  and  which  would,  via  the  MSC,  all  direct  inject  of  NSW  positions  to  the  COP.” 

o  Establish  a  resource  supply  link  with  forward  elements  to  feed  a  dynamic 
component  of  the  information  product. 

■  “Fuse  operations  and  intelligence,  located  in  the  same  physical  space  in  San  Diego,  to 
provide  the  same  staff  functions  without  a  large  deployed  footprint.” 

o  Take  advantage  of  the  location-independent  manufacturing  elements  and  shift  them 
to  appropriate  locations  in  the  supply  chain  that  minimize  the  costs  associated  with 
operating  these  elements. 
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8.  A  Concept  of  Information  Product  Reliability 


Every  information  product  consists  of  static  and  dynamic  components  linked  together  by  the  single 
purpose  motivating  its  manufacturing.  These  components,  both  in  presence  and  function,  have 
parallel  and  sequential  dependency  relationships  that  can  degrade,  fail,  be  refreshed,  and  can  be 
replaced.  In  this  sense,  the  notion  of  information  product  reliability  is  relevant.  Exactly  how  to 
define  these  relationships  and  what  underlying  mathematical  functions  describe  their  behavior  over 
time  is  our  ongoing  concern. 

In  this  vein,  we  believe  it  prudent  to  begin  to  construct  a  precise  foundation  for  reliability  theory  of 
an  information  product  from  first  principles  rather  than  simply  adopting  attractive  metaphors  where 
we  see  opportunity.  This  section  introduces  our  initial  thinking  on  the  matter  with  regards  to  a 
specific  information  product  being  manufactured  today:  digital  common  operating  pictures  (COP)  of 
the  battlespace  used  to  enhance  situational  awareness  of  friendly  ground  force  commanders  (and 
others). 

Having  previously  investigated  information  reliability  structures  for  sensor  systems  [7],  we  do  not 
concur  with  information  reliability  characterizations  based  simply  upon  propagated  uncertainty 
tracking.  Although  this  dimension  influences  what  we  would  traditionally  describe  when  discussing 
stochastic  measures  of  reliability,  focusing  on  this  aspect  in  the  context  of  manufacturing  information 
products  appears  to  be  of  little  use.  It  represents  an  aspect  of  the  information  manufacturing  system 
well  outside  the  span-of-control  of  product  design  and  manufacturing,  and  more  in  the  domain  of 
information  assurance  and  network  administrators  (thinking  in  a  digital  sense). 

This  earlier  probabilistic  structure  [7]  lines  up  with  Huang’s  [6]  systematic  approach,  which 
focused  on  how  information  may  become  deficient  during  the  information  production  process  though 
the  influences  of  uncertainty  generation.  At  the  heart  of  this  earlier  approach  is  a  presumption  that 
there  exists  a  perfect  data  state  against  which  any  product  could  be  compared.  Herein  lies  the 
difficulty  much  in  the  same  manner  that  caused  us  to  discount  the  usefulness  of  information  quality 
criteria  being  compared  to  ground  truth.  In  fact,  Eppler  [2]  notes  that  few  research  strategies  have 
adopted  this  deductive-analytic  or  ontological  approach  in  which  real-life  states  are  compared  to 
represented  data  states  for  many  of  the  same  reasons  we  posit. 

The  fundamental  characterization  of  information  reliability  we  are  exploring  is  based  on  a  “risk-to- 
use”  criterion  at  a  time  when  an  estimation  or  decision  must  be  made  that  is  directly  dependent  upon 
the  information  product  in  order  to  do  so.  A  military  information  product’s  manufacturing  design  is 
centered  on  a  comprehensive  understanding  of  how  the  product  is  going  to  be  used:  its  purpose.  This 
is  consistent  with  the  information  product  manufacturing  framework  we  present  herein.  Hence,  we 
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believe  that  since  this  purpose  binds  the  components  of  the  information  product,  it  therefore  is  the 
most  logical  construct  upon  which  to  develop  a  reliability  theory.  Furthermore,  since  we  assume  that 
purpose  follows  intent,  each  of  the  information  product’s  components  supporting  this  intent  is,  in 
effect,  providing  evidential  support  for  inference  estimation,  particularly  for  the  case  of  achieving 
situational  awareness  in  the  face  of  un-resolvable  uncertainty.  A  reasonable  challenge  is  imposed 
because  of  the  static  and  dynamic  nature  we  imbue  to  information  products. 

We  conjecture  that  exposure  to  some  subset  of  environmental  factors  affects  the  degree  of  support 
to  purpose  that  an  aggregated  information  product  provides.  Individual  components  can  fail  to  the 
extent  that  their  support  of  purpose  diminishes.  Towards  this  end,  each  of  the  components  has  an 
inherent  refresh  rate  (< dr/dt )  that  determines  the  extent  of  content  refreshing  and  the  time  interval 
within  which  such  refreshing  must  take  place  in  order  to  maintain  a  reliable  product.  These  must  also 
be  aligned  to  the  Little-q  deployment  schedule  associated  with  the  product. 

Past  a  threshold  level  of  minimum  support  for  purpose,  the  component  fails  to  support  the  purpose 
to  which  the  product  was  designed.  It  must  then  either  be  replaced,  discarded  as  a  component 
altogether,  or  be  refreshed  (which  is  equivalent  to  content  re-manufacturing,  the  idea  behind 
updating).  When  a  sufficient  number  of  components  fail,  mutual  evidential  support  for  the 
underlying  purpose  of  the  product  is  no  longer  present  and  the  information  product  as  a  whole  fails. 
Figure  3  illustrates  an  example  of  an  information  product  in  which  each  of  the  designated  areas 
contains  a  different  presentation  format,  content,  and  is  either  static  or  dynamic. 

It  is  possible  that  reliability  in  this  sense  could  be  inextricably  linked  to  the  end  user,  which  is  why 
investigating  this  idea  is  related  to  active/passive  user  interaction  with  digital  displays  currently  being 
investigated  by  the  Department  of  Behavioral  Sciences  at  USMA.  In  this  context,  each  product 
component  has  a  required  react  time  (At)  associated  with  it  that  captures  a  sense  of  the  component’s 
importance  to  the  purpose.  Components  with  small  At  are  of  high  importance,  those  with  large  At  are 
less  important.  In  fact,  the  relationship  of  a  component  to  the  product’s  purpose  in  this  manner 
defines  a  prioritized  dependency  relationship  that  could  drive  maintenance  and  security  planning  as 
well. 

The  failure  of  individual  components  can  be  described  by  either  deterministic  models  driven  by 
factors  under  the  control  of  product  manufacturers,  or  stochastic  models  driven  by  factors  not  under 
the  control  of  product  manufacturers.  The  latter  case  illustrates  an  interesting  dimension  to 
information  product  reliability:  that  it  can  fail  partially  or  completely  through  no  fault  of  its  own. 

And  this  characteristic  sets  it  distinctly  apart  from  other  applications  of  reliability. 

For  the  case  of  stochastic  models,  failure  distribution^*)  must  be  specified  and  validated.  Based 
on  the  component  failure  distributions  and  their  associated  relevancy  refresh  rates,  a  risk-to-use 
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measure  could  be  developed  akin  to  a  hazard  rate  function.  The  risk-to-use  measure  dynamically 
changes  with  time,  increasing  and  decreasing  as  maintenance  actions  take  place  and  the  content  and 
functions  of  the  product  are  updated. 

One  particular  purpose  for  information  products  is  to  support  decision  makers'  estimates  concerning 
OPFOR  operational  states  or  on  insurgent  effects  states.  This  is  support  for  inference.  If  reliability 
degraded,  the  product  or  its  components  ability  to  support  inference  degrades.  That  is,  given  that  H0 
is  false,  its  ability  to  support  Hi  with  tangible  evidence  presented  in  the  information  product 
diminishes.  One  could  in  this  case,  consider  the  mapping  from  evidence  to  inference  to  have  failed, 
which  would  go  to  assessing  the  reliability  of  the  mapping  function  itself,  or  that  a  single  component 
(or  more  than  one)  has  failed. 

We  suspect  that  our  notion  of  information  reliability  will  be  susceptible  to  modeling  using  a  quasi¬ 
renewal  process  based  on  the  distribution  of  inferential  support  of  purpose  conforming  to  an 
appropriate  functional  distribution  (See  Nachlas  et  al.  2004).  How  exactly  to  translate  this  and  what 
qualities  must  be  present  in  the  underlying  distribution  have  yet  to  be  answered. 
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9.  The  Kiviat  Diagram  as  an  Information  Product 


Kiviat  diagrams  have  been  used  for  many  years  in  computer  performance  evaluation  [46].  In 
general,  a  Kiviat  diagram  is  constructed  for  the  purpose  of  enabling  the  user  to  make  comparative 
assessments  cross  a  host  of  representative  dimensions  which,  in  aggregate,  represent  a  holistic 
evaluation  of  the  system  of  interest.  Each  value  of  each  measure  is  displayed  on  its  own  individual 
axis,  plotted  as  a  point.  By  connecting  the  points  across  each  axis,  a  pattern  is  formed. 

Consistent  with  this  purpose,  the  component  elements  presented  on  each  of  the  axes  are  mutually 
supporting.  When  the  display  is  static,  the  data  represents  the  results  of  assessment  “as  is.”  A  single 
webline  is  used  to  connect  the  points.  When  the  display  is  intentionally  dynamic,  attempting  to  enable 
the  user  to  perform  a  comparative  estimate  such  as  “before  and  after”,  then  multiple  weblines  are 
typically  used,  as  in  Figure  12. 

We  have  one  concern  with  the  use  of  Kiviat  diagrams  in  the  context  of  NCO  case  studies  that  we 
address,  that  being  scale  and  unit  differences  inducing  a  distorted  “A  effect.”  This  concern  arises 
because  of  our  sensitivity  to  the  role  of  a  Kiviat  diagram  as  an  information  product  when  it  is 
delivered  to  an  end  user. 

A.  Scale  and  unit  differences:  the  A  effect 

In  this  dynamic  context,  maintaining  normalized  axes  across  the  dimensions  is  critical  because  the 
marginal  differences,  say,  between  two  system’s  performance  criteria,  expressed  in  percentages  of 
differences  between  one  webline  (before)  and  another  (after),  are  both  meaningful  and  relevant.  When 
the  axes  have  different  scales  and  units,  a  reader  can  easily  be  misled  into  over-  or  underestimating 
the  comparative  differences  between  systems. 

Barzilai  ([50],  [51],  [52])  was  one  of  the  earliest  researchers  to  bring  this  concern  to  light.  At  the 
time,  his  concern  centered  on  the  lack  of  consistency  and  apparent  reversibility  of  results  when 
applying  an  Analytical  Hierarchy  Process  (AHP)  to  compare  multiple  criteria.  He  demonstrated 
mathematically  that  the  principle  reason  for  this  was  exactly  the  same  as  we  mention  here:  scale 
mismatch  and  scale  inadequacy. 

Unlike  Barzilai ’s  concern  over  reversibility  in  AHP,  our  concern  is  that  although  no  explicit 
mathematical  combination  of  scales  is  being  performed,  the  user  is  in  fact  performing  a  didactic 
functional  transformation  by  forming  opinions  and  drawing  conclusions  from  the  diagram.  Without 
explicit  statements  concerning  the  limitations  of  what  the  display  is  attempting  to  communicate  in 
comparison  between  systems,  a  reader  could  easily  misinterpret  limitations  as  to  what  these  should 
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be,  a  reader  could  easily  come  to  over  or  under  estimate  the  impact  of  system  differences. 


Comparing  MCPs  Using  Summary  Metrics 
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Figure  15.  Comparative  information  NCO  Kiviat  diagrams. 


Using  a  Kiviat  diagram  to  show  a  rollup  perspective  of  static  (single)  category  performance 
differences  is  not  advisable  unless  the  scales  used  on  the  individual  axes  are  identical.  The  Kiviat 
diagram  on  the  right  of  Figure  15  is  one  such  use  that  is  proper.  In  this  case,  subjective  data 
supporting  the  eight  categories  of  performance  were  gathered  using  survey  methods.  All  eight  axes 
scales  rely  on  a  single  Likert  scale  whose  values  run  between  1  and  5.  The  area  between  weblines 
then  properly  communicates  the  relative  proportional  differences  in  system  performance. 

One  must  still  be  careful  to  not  misuse  numerical  data  that  results  from  Likert  scales,  the  classic 
example  would  be  to  take  the  average  Likert  score  for,  say,  Degree  of  data  input,  and  use  it  to 
estimate  some  ‘population  mean’  judgment  concerning  the  same.  This  is  especially  a  concern  when 
one  transforms  subjective  data  onto  a  continuous  interval  scale,  as  in  the  Kiviat  diagram  on  the  left  in 
Figure  15. 

10.  Final  Note:  In  Defense  of  Generalizing  Case  Study  Results  to  Force 
Transformation  Theory 

It  is  important  to  note  the  logical  basis  for  generalizing  applications  of  information  quality  and 
service  reliability  concepts  to  the  NCOCF  using  the  case  studies  as  the  basis  of  application  and 
analysis.  Almost  without  exception,  each  of  the  various  case  studies  adds  caveat  addressing  the 
strength  of  evidence  in  support  of  transformation  lessons  to  wit:  the  small  size  of  the  sample  renders 
the  evidence  statistically  weak. 
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We  note  that  this  is  not  the  case  in  general,  and  that  the  intended  inferential  use  of  the  evidence 
resulting  from  interviews,  surveys,  and  direct  observation  sufficiently  justifies  its  purpose.  The  basis 
for  this  position 


Figure  16.  Yin’s  (1994)  concept  of  inferential  generalization. 

lies  in  the  distinction  between  the  strength  and  appropriate  use  of  observational  studies  versus 
controlled  studies. 

The  defense  of  using  single  or  multiple  case  studies  (in  any  event  small  sample  from  a  statistical 
sense)  is  laid  out  in  Lee  and  Baskerville  [7],  consistent  with  Yin  [8].  Figure  16  shows  Yin’s 
conceptualization  applied  to  the  NCOCF  case  study  research  effort.  The  upshot  of  their  position  is 
that  statistical  inference  from  a  sample  to  a  population  is  a  type  of  inductive  generalization  that 
cannot  be  improved  by  increasing  sample  size,  a  position  consistent  with  Hume’s  Truism  [12].  We 
agree. 

Increasing  sample  size  increases  the  reliability  of  the  statistical  estimate  because  of  strengthening 
the  underlying  mathematical  assumptions  upon  which  the  estimation  procedure  relies.  When  this  is 
possible,  the  statistical  estimate  validly  deduces  to  an  expectation  of  consistent  results  on  other 
identically  structured  random  samples  of  the  population,  but  not  to  generalizing  these  to  the 
population  at  large.  In  simple  terms,  while  increasing  the  sample  size  improves  the  reliability  of  a 
population  parameter  estimate,  it  does  nothing  to  either  strengthen  or  detract  from  the  inference  itself. 

Moreover,  specific  research  results  addressing  a  comparison  between  nonrandom  sample  based 
experiments  and  random  sample  based  experiments  ([13],  [14],  [15],  [16],  [17])  directly  support  this 
position. 

In  both  the  CTF-50  and  Stryker  case  studies,  analysts  emplaced  a  caveat  on  their  findings  of  a 
concern  that  their  surveys  were  applied  to  a  small,  but  representative  cross  section  of  the  NCO 
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information  network  users.  In  the  opinion  of  these  authors,  such  a  caveat  is  both  unnecessary  and 
unfounded  in  light  of  the  above.  In  both  cases,  survey  data  were  not  gathered  to  support  inference  to 
a  larger  population  of  users  nor  to  extrapolate  to  a  larger  population  of  NCO  systems  across  the 
services,  the  concern  of  statistics.  Inference  was  being  made  to  the  theory  of  NCO,  upon  which  the 
results,  even  as  a  small  sample,  stand  on  firm  ground. 
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